Compare commits

..

3 Commits

Author SHA1 Message Date
Lorenze Jay
b9dd166a6b Lorenze/agent executor flow pattern (#3975)
Some checks are pending
Build uv cache / build-cache (3.10) (push) Waiting to run
Build uv cache / build-cache (3.11) (push) Waiting to run
Build uv cache / build-cache (3.12) (push) Waiting to run
Build uv cache / build-cache (3.13) (push) Waiting to run
CodeQL Advanced / Analyze (actions) (push) Waiting to run
CodeQL Advanced / Analyze (python) (push) Waiting to run
Notify Downstream / notify-downstream (push) Waiting to run
* WIP gh pr refactor: update agent executor handling and introduce flow-based executor

* wip

* refactor: clean up comments and improve code clarity in agent executor flow

- Removed outdated comments and unnecessary explanations in  and  classes to enhance code readability.
- Simplified parameter updates in the agent executor to avoid confusion regarding executor recreation.
- Improved clarity in the  method to ensure proper handling of non-final answers without raising errors.

* bumping pytest-randomly numpy

* also bump versions of anthropic sdk

* ensure flow logs are not passed if its on executor

* revert anthropic bump

* fix

* refactor: update dependency markers in uv.lock for platform compatibility

- Enhanced dependency markers for , , , and others to ensure compatibility across different platforms (Linux, Darwin, and architecture-specific conditions).
- Removed unnecessary event emission in the  class during kickoff.
- Cleaned up commented-out code in the  class for better readability and maintainability.

* drop dupllicate

* test: enhance agent executor creation and stop word assertions

- Added calls to create_agent_executor in multiple test cases to ensure proper agent execution setup.
- Updated assertions for stop words in the agent tests to remove unnecessary checks and improve clarity.
- Ensured consistency in task handling by invoking create_agent_executor with the appropriate task parameter.

* refactor: reorganize agent executor imports and introduce CrewAgentExecutorFlow

- Removed the old import of CrewAgentExecutorFlow and replaced it with the new import from the experimental module.
- Updated relevant references in the codebase to ensure compatibility with the new structure.
- Enhanced the organization of imports in core.py and base_agent.py for better clarity and maintainability.

* updating name

* dropped usage of printer here for rich console and dropped non-added value logging

* address i18n

* Enhance concurrency control in CrewAgentExecutorFlow by introducing a threading lock to prevent concurrent executions. This change ensures that the executor instance cannot be invoked while already running, improving stability and reliability during flow execution.

* string literal returns

* string literal returns

* Enhance CrewAgentExecutor initialization by allowing optional i18n parameter for improved internationalization support. This change ensures that the executor can utilize a provided i18n instance or fallback to the default, enhancing flexibility in multilingual contexts.

---------

Co-authored-by: Greyson LaLonde <greyson.r.lalonde@gmail.com>
2025-12-28 10:21:32 -08:00
João Moura
c73b36a4c5 Adding HITL for Flows (#4143)
Some checks failed
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Check Documentation Broken Links / Check broken links (push) Has been cancelled
Notify Downstream / notify-downstream (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
* feat: introduce human feedback events and decorator for flow methods

- Added HumanFeedbackRequestedEvent and HumanFeedbackReceivedEvent classes to handle human feedback interactions within flows.
- Implemented the @human_feedback decorator to facilitate human-in-the-loop workflows, allowing for feedback collection and routing based on responses.
- Enhanced Flow class to store human feedback history and manage feedback outcomes.
- Updated flow wrappers to preserve attributes from methods decorated with @human_feedback.
- Added integration and unit tests for the new human feedback functionality, ensuring proper validation and routing behavior.

* adding deployment docs

* New docs

* fix printer

* wrong change

* Adding Async Support
feat: enhance human feedback support in flows

- Updated the @human_feedback decorator to use 'message' parameter instead of 'request' for clarity.
- Introduced new FlowPausedEvent and MethodExecutionPausedEvent to handle flow and method pauses during human feedback.
- Added ConsoleProvider for synchronous feedback collection and integrated async feedback capabilities.
- Implemented SQLite persistence for managing pending feedback context.
- Expanded documentation to include examples of async human feedback usage and best practices.

* linter

* fix

* migrating off printer

* updating docs

* new tests

* doc update
2025-12-25 21:04:10 -03:00
Lucas Gomide
0c020991c4 docs: fix wrong trigger name in sample docs (#4147)
Some checks failed
Check Documentation Broken Links / Check broken links (push) Has been cancelled
Notify Downstream / notify-downstream (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Mark stale issues and pull requests / stale (push) Has been cancelled
Build uv cache / build-cache (3.10) (push) Has been cancelled
Build uv cache / build-cache (3.11) (push) Has been cancelled
Build uv cache / build-cache (3.12) (push) Has been cancelled
Build uv cache / build-cache (3.13) (push) Has been cancelled
2025-12-23 08:41:51 -05:00
43 changed files with 7710 additions and 1021 deletions

View File

@@ -308,6 +308,7 @@
"en/learn/hierarchical-process",
"en/learn/human-input-on-execution",
"en/learn/human-in-the-loop",
"en/learn/human-feedback-in-flows",
"en/learn/kickoff-async",
"en/learn/kickoff-for-each",
"en/learn/llm-connections",
@@ -735,6 +736,7 @@
"pt-BR/learn/hierarchical-process",
"pt-BR/learn/human-input-on-execution",
"pt-BR/learn/human-in-the-loop",
"pt-BR/learn/human-feedback-in-flows",
"pt-BR/learn/kickoff-async",
"pt-BR/learn/kickoff-for-each",
"pt-BR/learn/llm-connections",
@@ -1171,6 +1173,7 @@
"ko/learn/hierarchical-process",
"ko/learn/human-input-on-execution",
"ko/learn/human-in-the-loop",
"ko/learn/human-feedback-in-flows",
"ko/learn/kickoff-async",
"ko/learn/kickoff-for-each",
"ko/learn/llm-connections",

View File

@@ -572,6 +572,55 @@ The `third_method` and `fourth_method` listen to the output of the `second_metho
When you run this Flow, the output will change based on the random boolean value generated by the `start_method`.
### Human in the Loop (human feedback)
The `@human_feedback` decorator enables human-in-the-loop workflows by pausing flow execution to collect feedback from a human. This is useful for approval gates, quality review, and decision points that require human judgment.
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Do you approve this content?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def generate_content(self):
return "Content to be reviewed..."
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"Approved! Feedback: {result.feedback}")
@listen("rejected")
def on_rejection(self, result: HumanFeedbackResult):
print(f"Rejected. Reason: {result.feedback}")
```
When `emit` is specified, the human's free-form feedback is interpreted by an LLM and collapsed into one of the specified outcomes, which then triggers the corresponding `@listen` decorator.
You can also use `@human_feedback` without routing to simply collect feedback:
```python Code
@start()
@human_feedback(message="Any comments on this output?")
def my_method(self):
return "Output for review"
@listen(my_method)
def next_step(self, result: HumanFeedbackResult):
# Access feedback via result.feedback
# Access original output via result.output
pass
```
Access all feedback collected during a flow via `self.last_human_feedback` (most recent) or `self.human_feedback_history` (all feedback as a list).
For a complete guide on human feedback in flows, including **async/non-blocking feedback** with custom providers (Slack, webhooks, etc.), see [Human Feedback in Flows](/en/learn/human-feedback-in-flows).
## Adding Agents to Flows
Agents can be seamlessly integrated into your flows, providing a lightweight alternative to full Crews when you need simpler, focused task execution. Here's an example of how to use an Agent within a flow to perform market research:

View File

@@ -62,13 +62,13 @@ Test your Gmail trigger integration locally using the CrewAI CLI:
crewai triggers list
# Simulate a Gmail trigger with realistic payload
crewai triggers run gmail/new_email
crewai triggers run gmail/new_email_received
```
The `crewai triggers run` command will execute your crew with a complete Gmail payload, allowing you to test your parsing logic before deployment.
<Warning>
Use `crewai triggers run gmail/new_email` (not `crewai run`) to simulate trigger execution during development. After deployment, your crew will automatically receive the trigger payload.
Use `crewai triggers run gmail/new_email_received` (not `crewai run`) to simulate trigger execution during development. After deployment, your crew will automatically receive the trigger payload.
</Warning>
## Monitoring Executions
@@ -83,6 +83,6 @@ Track history and performance of triggered runs:
- Ensure Gmail is connected in Tools & Integrations
- Verify the Gmail Trigger is enabled on the Triggers tab
- Test locally with `crewai triggers run gmail/new_email` to see the exact payload structure
- Test locally with `crewai triggers run gmail/new_email_received` to see the exact payload structure
- Check the execution logs and confirm the payload is passed as `crewai_trigger_payload`
- Remember: use `crewai triggers run` (not `crewai run`) to simulate trigger execution

View File

@@ -0,0 +1,581 @@
---
title: Human Feedback in Flows
description: Learn how to integrate human feedback directly into your CrewAI Flows using the @human_feedback decorator
icon: user-check
mode: "wide"
---
## Overview
The `@human_feedback` decorator enables human-in-the-loop (HITL) workflows directly within CrewAI Flows. It allows you to pause flow execution, present output to a human for review, collect their feedback, and optionally route to different listeners based on the feedback outcome.
This is particularly valuable for:
- **Quality assurance**: Review AI-generated content before it's used downstream
- **Decision gates**: Let humans make critical decisions in automated workflows
- **Approval workflows**: Implement approve/reject/revise patterns
- **Interactive refinement**: Collect feedback to improve outputs iteratively
```mermaid
flowchart LR
A[Flow Method] --> B[Output Generated]
B --> C[Human Reviews]
C --> D{Feedback}
D -->|emit specified| E[LLM Collapses to Outcome]
D -->|no emit| F[HumanFeedbackResult]
E --> G["@listen('approved')"]
E --> H["@listen('rejected')"]
F --> I[Next Listener]
```
## Quick Start
Here's the simplest way to add human feedback to a flow:
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback
class SimpleReviewFlow(Flow):
@start()
@human_feedback(message="Please review this content:")
def generate_content(self):
return "This is AI-generated content that needs review."
@listen(generate_content)
def process_feedback(self, result):
print(f"Content: {result.output}")
print(f"Human said: {result.feedback}")
flow = SimpleReviewFlow()
flow.kickoff()
```
When this flow runs, it will:
1. Execute `generate_content` and return the string
2. Display the output to the user with the request message
3. Wait for the user to type feedback (or press Enter to skip)
4. Pass a `HumanFeedbackResult` object to `process_feedback`
## The @human_feedback Decorator
### Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `message` | `str` | Yes | The message shown to the human alongside the method output |
| `emit` | `Sequence[str]` | No | List of possible outcomes. Feedback is collapsed to one of these, which triggers `@listen` decorators |
| `llm` | `str \| BaseLLM` | When `emit` specified | LLM used to interpret feedback and map to an outcome |
| `default_outcome` | `str` | No | Outcome to use if no feedback provided. Must be in `emit` |
| `metadata` | `dict` | No | Additional data for enterprise integrations |
| `provider` | `HumanFeedbackProvider` | No | Custom provider for async/non-blocking feedback. See [Async Human Feedback](#async-human-feedback-non-blocking) |
### Basic Usage (No Routing)
When you don't specify `emit`, the decorator simply collects feedback and passes a `HumanFeedbackResult` to the next listener:
```python Code
@start()
@human_feedback(message="What do you think of this analysis?")
def analyze_data(self):
return "Analysis results: Revenue up 15%, costs down 8%"
@listen(analyze_data)
def handle_feedback(self, result):
# result is a HumanFeedbackResult
print(f"Analysis: {result.output}")
print(f"Feedback: {result.feedback}")
```
### Routing with emit
When you specify `emit`, the decorator becomes a router. The human's free-form feedback is interpreted by an LLM and collapsed into one of the specified outcomes:
```python Code
@start()
@human_feedback(
message="Do you approve this content for publication?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_content(self):
return "Draft blog post content here..."
@listen("approved")
def publish(self, result):
print(f"Publishing! User said: {result.feedback}")
@listen("rejected")
def discard(self, result):
print(f"Discarding. Reason: {result.feedback}")
@listen("needs_revision")
def revise(self, result):
print(f"Revising based on: {result.feedback}")
```
<Tip>
The LLM uses structured outputs (function calling) when available to guarantee the response is one of your specified outcomes. This makes routing reliable and predictable.
</Tip>
## HumanFeedbackResult
The `HumanFeedbackResult` dataclass contains all information about a human feedback interaction:
```python Code
from crewai.flow.human_feedback import HumanFeedbackResult
@dataclass
class HumanFeedbackResult:
output: Any # The original method output shown to the human
feedback: str # The raw feedback text from the human
outcome: str | None # The collapsed outcome (if emit was specified)
timestamp: datetime # When the feedback was received
method_name: str # Name of the decorated method
metadata: dict # Any metadata passed to the decorator
```
### Accessing in Listeners
When a listener is triggered by a `@human_feedback` method with `emit`, it receives the `HumanFeedbackResult`:
```python Code
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"Original output: {result.output}")
print(f"User feedback: {result.feedback}")
print(f"Outcome: {result.outcome}") # "approved"
print(f"Received at: {result.timestamp}")
```
## Accessing Feedback History
The `Flow` class provides two attributes for accessing human feedback:
### last_human_feedback
Returns the most recent `HumanFeedbackResult`:
```python Code
@listen(some_method)
def check_feedback(self):
if self.last_human_feedback:
print(f"Last feedback: {self.last_human_feedback.feedback}")
```
### human_feedback_history
A list of all `HumanFeedbackResult` objects collected during the flow:
```python Code
@listen(final_step)
def summarize(self):
print(f"Total feedback collected: {len(self.human_feedback_history)}")
for i, fb in enumerate(self.human_feedback_history):
print(f"{i+1}. {fb.method_name}: {fb.outcome or 'no routing'}")
```
<Warning>
Each `HumanFeedbackResult` is appended to `human_feedback_history`, so multiple feedback steps won't overwrite each other. Use this list to access all feedback collected during the flow.
</Warning>
## Complete Example: Content Approval Workflow
Here's a full example implementing a content review and approval workflow:
<CodeGroup>
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
from pydantic import BaseModel
class ContentState(BaseModel):
topic: str = ""
draft: str = ""
final_content: str = ""
revision_count: int = 0
class ContentApprovalFlow(Flow[ContentState]):
"""A flow that generates content and gets human approval."""
@start()
def get_topic(self):
self.state.topic = input("What topic should I write about? ")
return self.state.topic
@listen(get_topic)
def generate_draft(self, topic):
# In real use, this would call an LLM
self.state.draft = f"# {topic}\n\nThis is a draft about {topic}..."
return self.state.draft
@listen(generate_draft)
@human_feedback(
message="Please review this draft. Reply 'approved', 'rejected', or provide revision feedback:",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_draft(self, draft):
return draft
@listen("approved")
def publish_content(self, result: HumanFeedbackResult):
self.state.final_content = result.output
print("\n✅ Content approved and published!")
print(f"Reviewer comment: {result.feedback}")
return "published"
@listen("rejected")
def handle_rejection(self, result: HumanFeedbackResult):
print("\n❌ Content rejected")
print(f"Reason: {result.feedback}")
return "rejected"
@listen("needs_revision")
def revise_content(self, result: HumanFeedbackResult):
self.state.revision_count += 1
print(f"\n📝 Revision #{self.state.revision_count} requested")
print(f"Feedback: {result.feedback}")
# In a real flow, you might loop back to generate_draft
# For this example, we just acknowledge
return "revision_requested"
# Run the flow
flow = ContentApprovalFlow()
result = flow.kickoff()
print(f"\nFlow completed. Revisions requested: {flow.state.revision_count}")
```
```text Output
What topic should I write about? AI Safety
==================================================
OUTPUT FOR REVIEW:
==================================================
# AI Safety
This is a draft about AI Safety...
==================================================
Please review this draft. Reply 'approved', 'rejected', or provide revision feedback:
(Press Enter to skip, or type your feedback)
Your feedback: Looks good, approved!
✅ Content approved and published!
Reviewer comment: Looks good, approved!
Flow completed. Revisions requested: 0
```
</CodeGroup>
## Combining with Other Decorators
The `@human_feedback` decorator works with other flow decorators. Place it as the innermost decorator (closest to the function):
```python Code
# Correct: @human_feedback is innermost (closest to the function)
@start()
@human_feedback(message="Review this:")
def my_start_method(self):
return "content"
@listen(other_method)
@human_feedback(message="Review this too:")
def my_listener(self, data):
return f"processed: {data}"
```
<Tip>
Place `@human_feedback` as the innermost decorator (last/closest to the function) so it wraps the method directly and can capture the return value before passing to the flow system.
</Tip>
## Best Practices
### 1. Write Clear Request Messages
The `request` parameter is what the human sees. Make it actionable:
```python Code
# ✅ Good - clear and actionable
@human_feedback(message="Does this summary accurately capture the key points? Reply 'yes' or explain what's missing:")
# ❌ Bad - vague
@human_feedback(message="Review this:")
```
### 2. Choose Meaningful Outcomes
When using `emit`, pick outcomes that map naturally to human responses:
```python Code
# ✅ Good - natural language outcomes
emit=["approved", "rejected", "needs_more_detail"]
# ❌ Bad - technical or unclear
emit=["state_1", "state_2", "state_3"]
```
### 3. Always Provide a Default Outcome
Use `default_outcome` to handle cases where users press Enter without typing:
```python Code
@human_feedback(
message="Approve? (press Enter to request revision)",
emit=["approved", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision", # Safe default
)
```
### 4. Use Feedback History for Audit Trails
Access `human_feedback_history` to create audit logs:
```python Code
@listen(final_step)
def create_audit_log(self):
log = []
for fb in self.human_feedback_history:
log.append({
"step": fb.method_name,
"outcome": fb.outcome,
"feedback": fb.feedback,
"timestamp": fb.timestamp.isoformat(),
})
return log
```
### 5. Handle Both Routed and Non-Routed Feedback
When designing flows, consider whether you need routing:
| Scenario | Use |
|----------|-----|
| Simple review, just need the feedback text | No `emit` |
| Need to branch to different paths based on response | Use `emit` |
| Approval gates with approve/reject/revise | Use `emit` |
| Collecting comments for logging only | No `emit` |
## Async Human Feedback (Non-Blocking)
By default, `@human_feedback` blocks execution waiting for console input. For production applications, you may need **async/non-blocking** feedback that integrates with external systems like Slack, email, webhooks, or APIs.
### The Provider Abstraction
Use the `provider` parameter to specify a custom feedback collection strategy:
```python Code
from crewai.flow import Flow, start, human_feedback, HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
class WebhookProvider(HumanFeedbackProvider):
"""Provider that pauses flow and waits for webhook callback."""
def __init__(self, webhook_url: str):
self.webhook_url = webhook_url
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# Notify external system (e.g., send Slack message, create ticket)
self.send_notification(context)
# Pause execution - framework handles persistence automatically
raise HumanFeedbackPending(
context=context,
callback_info={"webhook_url": f"{self.webhook_url}/{context.flow_id}"}
)
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Review this content:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=WebhookProvider("https://myapp.com/api"),
)
def generate_content(self):
return "AI-generated content..."
@listen("approved")
def publish(self, result):
return "Published!"
```
<Tip>
The flow framework **automatically persists state** when `HumanFeedbackPending` is raised. Your provider only needs to notify the external system and raise the exception—no manual persistence calls required.
</Tip>
### Handling Paused Flows
When using an async provider, `kickoff()` returns a `HumanFeedbackPending` object instead of raising an exception:
```python Code
flow = ReviewFlow()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
# Flow is paused, state is automatically persisted
print(f"Waiting for feedback at: {result.callback_info['webhook_url']}")
print(f"Flow ID: {result.context.flow_id}")
else:
# Normal completion
print(f"Flow completed: {result}")
```
### Resuming a Paused Flow
When feedback arrives (e.g., via webhook), resume the flow:
```python Code
# Sync handler:
def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = flow.resume(feedback)
return result
# Async handler (FastAPI, aiohttp, etc.):
async def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = await flow.resume_async(feedback)
return result
```
### Key Types
| Type | Description |
|------|-------------|
| `HumanFeedbackProvider` | Protocol for custom feedback providers |
| `PendingFeedbackContext` | Contains all info needed to resume a paused flow |
| `HumanFeedbackPending` | Returned by `kickoff()` when flow is paused for feedback |
| `ConsoleProvider` | Default blocking console input provider |
### PendingFeedbackContext
The context contains everything needed to resume:
```python Code
@dataclass
class PendingFeedbackContext:
flow_id: str # Unique identifier for this flow execution
flow_class: str # Fully qualified class name
method_name: str # Method that triggered feedback
method_output: Any # Output shown to the human
message: str # The request message
emit: list[str] | None # Possible outcomes for routing
default_outcome: str | None
metadata: dict # Custom metadata
llm: str | None # LLM for outcome collapsing
requested_at: datetime
```
### Complete Async Flow Example
```python Code
from crewai.flow import (
Flow, start, listen, human_feedback,
HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
)
class SlackNotificationProvider(HumanFeedbackProvider):
"""Provider that sends Slack notifications and pauses for async feedback."""
def __init__(self, channel: str):
self.channel = channel
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# Send Slack notification (implement your own)
slack_thread_id = self.post_to_slack(
channel=self.channel,
message=f"Review needed:\n\n{context.method_output}\n\n{context.message}",
)
# Pause execution - framework handles persistence automatically
raise HumanFeedbackPending(
context=context,
callback_info={
"slack_channel": self.channel,
"thread_id": slack_thread_id,
}
)
class ContentPipeline(Flow):
@start()
@human_feedback(
message="Approve this content for publication?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
provider=SlackNotificationProvider("#content-reviews"),
)
def generate_content(self):
return "AI-generated blog post content..."
@listen("approved")
def publish(self, result):
print(f"Publishing! Reviewer said: {result.feedback}")
return {"status": "published"}
@listen("rejected")
def archive(self, result):
print(f"Archived. Reason: {result.feedback}")
return {"status": "archived"}
@listen("needs_revision")
def queue_revision(self, result):
print(f"Queued for revision: {result.feedback}")
return {"status": "revision_needed"}
# Starting the flow (will pause and wait for Slack response)
def start_content_pipeline():
flow = ContentPipeline()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
return {"status": "pending", "flow_id": result.context.flow_id}
return result
# Resuming when Slack webhook fires (sync handler)
def on_slack_feedback(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = flow.resume(slack_message)
return result
# If your handler is async (FastAPI, aiohttp, Slack Bolt async, etc.)
async def on_slack_feedback_async(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = await flow.resume_async(slack_message)
return result
```
<Warning>
If you're using an async web framework (FastAPI, aiohttp, Slack Bolt async mode), use `await flow.resume_async()` instead of `flow.resume()`. Calling `resume()` from within a running event loop will raise a `RuntimeError`.
</Warning>
### Best Practices for Async Feedback
1. **Check the return type**: `kickoff()` returns `HumanFeedbackPending` when paused—no try/except needed
2. **Use the right resume method**: Use `resume()` in sync code, `await resume_async()` in async code
3. **Store callback info**: Use `callback_info` to store webhook URLs, ticket IDs, etc.
4. **Implement idempotency**: Your resume handler should be idempotent for safety
5. **Automatic persistence**: State is automatically saved when `HumanFeedbackPending` is raised and uses `SQLiteFlowPersistence` by default
6. **Custom persistence**: Pass a custom persistence instance to `from_pending()` if needed
## Related Documentation
- [Flows Overview](/en/concepts/flows) - Learn about CrewAI Flows
- [Flow State Management](/en/guides/flows/mastering-flow-state) - Managing state in flows
- [Flow Persistence](/en/concepts/flows#persistence) - Persisting flow state
- [Routing with @router](/en/concepts/flows#router) - More about conditional routing
- [Human Input on Execution](/en/learn/human-input-on-execution) - Task-level human input

View File

@@ -5,9 +5,22 @@ icon: "user-check"
mode: "wide"
---
Human-in-the-Loop (HITL) is a powerful approach that combines artificial intelligence with human expertise to enhance decision-making and improve task outcomes. This guide shows you how to implement HITL within CrewAI.
Human-in-the-Loop (HITL) is a powerful approach that combines artificial intelligence with human expertise to enhance decision-making and improve task outcomes. CrewAI provides multiple ways to implement HITL depending on your needs.
## Setting Up HITL Workflows
## Choosing Your HITL Approach
CrewAI offers two main approaches for implementing human-in-the-loop workflows:
| Approach | Best For | Integration |
|----------|----------|-------------|
| **Flow-based** (`@human_feedback` decorator) | Local development, console-based review, synchronous workflows | [Human Feedback in Flows](/en/learn/human-feedback-in-flows) |
| **Webhook-based** (Enterprise) | Production deployments, async workflows, external integrations (Slack, Teams, etc.) | This guide |
<Tip>
If you're building flows and want to add human review steps with routing based on feedback, check out the [Human Feedback in Flows](/en/learn/human-feedback-in-flows) guide for the `@human_feedback` decorator.
</Tip>
## Setting Up Webhook-Based HITL Workflows
<Steps>
<Step title="Configure Your Task">

View File

@@ -565,6 +565,55 @@ Fourth method running
이 Flow를 실행하면, `start_method`에서 생성된 랜덤 불리언 값에 따라 출력값이 달라집니다.
### Human in the Loop (인간 피드백)
`@human_feedback` 데코레이터는 인간의 피드백을 수집하기 위해 플로우 실행을 일시 중지하는 human-in-the-loop 워크플로우를 가능하게 합니다. 이는 승인 게이트, 품질 검토, 인간의 판단이 필요한 결정 지점에 유용합니다.
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
class ReviewFlow(Flow):
@start()
@human_feedback(
message="이 콘텐츠를 승인하시겠습니까?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def generate_content(self):
return "검토할 콘텐츠..."
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"승인됨! 피드백: {result.feedback}")
@listen("rejected")
def on_rejection(self, result: HumanFeedbackResult):
print(f"거부됨. 이유: {result.feedback}")
```
`emit`이 지정되면, 인간의 자유 형식 피드백이 LLM에 의해 해석되어 지정된 outcome 중 하나로 매핑되고, 해당 `@listen` 데코레이터를 트리거합니다.
라우팅 없이 단순히 피드백만 수집할 수도 있습니다:
```python Code
@start()
@human_feedback(message="이 출력에 대한 코멘트가 있으신가요?")
def my_method(self):
return "검토할 출력"
@listen(my_method)
def next_step(self, result: HumanFeedbackResult):
# result.feedback로 피드백에 접근
# result.output으로 원래 출력에 접근
pass
```
플로우 실행 중 수집된 모든 피드백은 `self.last_human_feedback` (가장 최근) 또는 `self.human_feedback_history` (리스트 형태의 모든 피드백)를 통해 접근할 수 있습니다.
플로우에서의 인간 피드백에 대한 완전한 가이드는 비동기/논블로킹 피드백과 커스텀 프로바이더(Slack, 웹훅 등)를 포함하여 [Flow에서 인간 피드백](/ko/learn/human-feedback-in-flows)을 참조하세요.
## 플로우에 에이전트 추가하기
에이전트는 플로우에 원활하게 통합할 수 있으며, 단순하고 집중된 작업 실행이 필요할 때 전체 Crew의 경량 대안으로 활용됩니다. 아래는 에이전트를 플로우 내에서 사용하여 시장 조사를 수행하는 예시입니다:

View File

@@ -62,13 +62,13 @@ CrewAI CLI를 사용하여 Gmail 트리거 통합을 로컬에서 테스트하
crewai triggers list
# 실제 payload로 Gmail 트리거 시뮬레이션
crewai triggers run gmail/new_email
crewai triggers run gmail/new_email_received
```
`crewai triggers run` 명령은 완전한 Gmail payload로 크루를 실행하여 배포 전에 파싱 로직을 테스트할 수 있게 해줍니다.
<Warning>
개발 중에는 `crewai triggers run gmail/new_email`을 사용하세요 (`crewai run`이 아님). 배포 후에는 크루가 자동으로 트리거 payload를 받습니다.
개발 중에는 `crewai triggers run gmail/new_email_received`을 사용하세요 (`crewai run`이 아님). 배포 후에는 크루가 자동으로 트리거 payload를 받습니다.
</Warning>
## Monitoring Executions
@@ -83,6 +83,6 @@ Track history and performance of triggered runs:
- Ensure Gmail is connected in Tools & Integrations
- Verify the Gmail Trigger is enabled on the Triggers tab
- `crewai triggers run gmail/new_email`로 로컬 테스트하여 정확한 payload 구조를 확인하세요
- `crewai triggers run gmail/new_email_received`로 로컬 테스트하여 정확한 payload 구조를 확인하세요
- Check the execution logs and confirm the payload is passed as `crewai_trigger_payload`
- 주의: 트리거 실행을 시뮬레이션하려면 `crewai triggers run`을 사용하세요 (`crewai run`이 아님)

View File

@@ -0,0 +1,581 @@
---
title: Flow에서 인간 피드백
description: "@human_feedback 데코레이터를 사용하여 CrewAI Flow에 인간 피드백을 직접 통합하는 방법을 알아보세요"
icon: user-check
mode: "wide"
---
## 개요
`@human_feedback` 데코레이터는 CrewAI Flow 내에서 직접 human-in-the-loop(HITL) 워크플로우를 가능하게 합니다. Flow 실행을 일시 중지하고, 인간에게 검토를 위해 출력을 제시하고, 피드백을 수집하고, 선택적으로 피드백 결과에 따라 다른 리스너로 라우팅할 수 있습니다.
이는 특히 다음과 같은 경우에 유용합니다:
- **품질 보증**: AI가 생성한 콘텐츠를 다운스트림에서 사용하기 전에 검토
- **결정 게이트**: 자동화된 워크플로우에서 인간이 중요한 결정을 내리도록 허용
- **승인 워크플로우**: 승인/거부/수정 패턴 구현
- **대화형 개선**: 출력을 반복적으로 개선하기 위해 피드백 수집
```mermaid
flowchart LR
A[Flow 메서드] --> B[출력 생성됨]
B --> C[인간이 검토]
C --> D{피드백}
D -->|emit 지정됨| E[LLM이 Outcome으로 매핑]
D -->|emit 없음| F[HumanFeedbackResult]
E --> G["@listen('approved')"]
E --> H["@listen('rejected')"]
F --> I[다음 리스너]
```
## 빠른 시작
Flow에 인간 피드백을 추가하는 가장 간단한 방법은 다음과 같습니다:
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback
class SimpleReviewFlow(Flow):
@start()
@human_feedback(message="이 콘텐츠를 검토해 주세요:")
def generate_content(self):
return "검토가 필요한 AI 생성 콘텐츠입니다."
@listen(generate_content)
def process_feedback(self, result):
print(f"콘텐츠: {result.output}")
print(f"인간의 의견: {result.feedback}")
flow = SimpleReviewFlow()
flow.kickoff()
```
이 Flow를 실행하면:
1. `generate_content`를 실행하고 문자열을 반환합니다
2. 요청 메시지와 함께 사용자에게 출력을 표시합니다
3. 사용자가 피드백을 입력할 때까지 대기합니다 (또는 Enter를 눌러 건너뜁니다)
4. `HumanFeedbackResult` 객체를 `process_feedback`에 전달합니다
## @human_feedback 데코레이터
### 매개변수
| 매개변수 | 타입 | 필수 | 설명 |
|----------|------|------|------|
| `message` | `str` | 예 | 메서드 출력과 함께 인간에게 표시되는 메시지 |
| `emit` | `Sequence[str]` | 아니오 | 가능한 outcome 목록. 피드백이 이 중 하나로 매핑되어 `@listen` 데코레이터를 트리거합니다 |
| `llm` | `str \| BaseLLM` | `emit` 지정 시 | 피드백을 해석하고 outcome에 매핑하는 데 사용되는 LLM |
| `default_outcome` | `str` | 아니오 | 피드백이 제공되지 않을 때 사용할 outcome. `emit`에 있어야 합니다 |
| `metadata` | `dict` | 아니오 | 엔터프라이즈 통합을 위한 추가 데이터 |
| `provider` | `HumanFeedbackProvider` | 아니오 | 비동기/논블로킹 피드백을 위한 커스텀 프로바이더. [비동기 인간 피드백](#비동기-인간-피드백-논블로킹) 참조 |
### 기본 사용법 (라우팅 없음)
`emit`을 지정하지 않으면, 데코레이터는 단순히 피드백을 수집하고 다음 리스너에 `HumanFeedbackResult`를 전달합니다:
```python Code
@start()
@human_feedback(message="이 분석에 대해 어떻게 생각하시나요?")
def analyze_data(self):
return "분석 결과: 매출 15% 증가, 비용 8% 감소"
@listen(analyze_data)
def handle_feedback(self, result):
# result는 HumanFeedbackResult입니다
print(f"분석: {result.output}")
print(f"피드백: {result.feedback}")
```
### emit을 사용한 라우팅
`emit`을 지정하면, 데코레이터는 라우터가 됩니다. 인간의 자유 형식 피드백이 LLM에 의해 해석되어 지정된 outcome 중 하나로 매핑됩니다:
```python Code
@start()
@human_feedback(
message="이 콘텐츠의 출판을 승인하시겠습니까?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_content(self):
return "블로그 게시물 초안 내용..."
@listen("approved")
def publish(self, result):
print(f"출판 중! 사용자 의견: {result.feedback}")
@listen("rejected")
def discard(self, result):
print(f"폐기됨. 이유: {result.feedback}")
@listen("needs_revision")
def revise(self, result):
print(f"다음을 기반으로 수정 중: {result.feedback}")
```
<Tip>
LLM은 가능한 경우 구조화된 출력(function calling)을 사용하여 응답이 지정된 outcome 중 하나임을 보장합니다. 이로 인해 라우팅이 신뢰할 수 있고 예측 가능해집니다.
</Tip>
## HumanFeedbackResult
`HumanFeedbackResult` 데이터클래스는 인간 피드백 상호작용에 대한 모든 정보를 포함합니다:
```python Code
from crewai.flow.human_feedback import HumanFeedbackResult
@dataclass
class HumanFeedbackResult:
output: Any # 인간에게 표시된 원래 메서드 출력
feedback: str # 인간의 원시 피드백 텍스트
outcome: str | None # 매핑된 outcome (emit이 지정된 경우)
timestamp: datetime # 피드백이 수신된 시간
method_name: str # 데코레이터된 메서드의 이름
metadata: dict # 데코레이터에 전달된 모든 메타데이터
```
### 리스너에서 접근하기
`emit`이 있는 `@human_feedback` 메서드에 의해 리스너가 트리거되면, `HumanFeedbackResult`를 받습니다:
```python Code
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"원래 출력: {result.output}")
print(f"사용자 피드백: {result.feedback}")
print(f"Outcome: {result.outcome}") # "approved"
print(f"수신 시간: {result.timestamp}")
```
## 피드백 히스토리 접근하기
`Flow` 클래스는 인간 피드백에 접근하기 위한 두 가지 속성을 제공합니다:
### last_human_feedback
가장 최근의 `HumanFeedbackResult`를 반환합니다:
```python Code
@listen(some_method)
def check_feedback(self):
if self.last_human_feedback:
print(f"마지막 피드백: {self.last_human_feedback.feedback}")
```
### human_feedback_history
Flow 동안 수집된 모든 `HumanFeedbackResult` 객체의 리스트입니다:
```python Code
@listen(final_step)
def summarize(self):
print(f"수집된 총 피드백: {len(self.human_feedback_history)}")
for i, fb in enumerate(self.human_feedback_history):
print(f"{i+1}. {fb.method_name}: {fb.outcome or '라우팅 없음'}")
```
<Warning>
각 `HumanFeedbackResult`는 `human_feedback_history`에 추가되므로, 여러 피드백 단계가 서로 덮어쓰지 않습니다. 이 리스트를 사용하여 Flow 동안 수집된 모든 피드백에 접근하세요.
</Warning>
## 완전한 예제: 콘텐츠 승인 워크플로우
콘텐츠 검토 및 승인 워크플로우를 구현하는 전체 예제입니다:
<CodeGroup>
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
from pydantic import BaseModel
class ContentState(BaseModel):
topic: str = ""
draft: str = ""
final_content: str = ""
revision_count: int = 0
class ContentApprovalFlow(Flow[ContentState]):
"""콘텐츠를 생성하고 인간의 승인을 받는 Flow입니다."""
@start()
def get_topic(self):
self.state.topic = input("어떤 주제에 대해 글을 쓸까요? ")
return self.state.topic
@listen(get_topic)
def generate_draft(self, topic):
# 실제 사용에서는 LLM을 호출합니다
self.state.draft = f"# {topic}\n\n{topic}에 대한 초안입니다..."
return self.state.draft
@listen(generate_draft)
@human_feedback(
message="이 초안을 검토해 주세요. 'approved', 'rejected'로 답하거나 수정 피드백을 제공해 주세요:",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_draft(self, draft):
return draft
@listen("approved")
def publish_content(self, result: HumanFeedbackResult):
self.state.final_content = result.output
print("\n✅ 콘텐츠가 승인되어 출판되었습니다!")
print(f"검토자 코멘트: {result.feedback}")
return "published"
@listen("rejected")
def handle_rejection(self, result: HumanFeedbackResult):
print("\n❌ 콘텐츠가 거부되었습니다")
print(f"이유: {result.feedback}")
return "rejected"
@listen("needs_revision")
def revise_content(self, result: HumanFeedbackResult):
self.state.revision_count += 1
print(f"\n📝 수정 #{self.state.revision_count} 요청됨")
print(f"피드백: {result.feedback}")
# 실제 Flow에서는 generate_draft로 돌아갈 수 있습니다
# 이 예제에서는 단순히 확인합니다
return "revision_requested"
# Flow 실행
flow = ContentApprovalFlow()
result = flow.kickoff()
print(f"\nFlow 완료. 요청된 수정: {flow.state.revision_count}")
```
```text Output
어떤 주제에 대해 글을 쓸까요? AI 안전
==================================================
OUTPUT FOR REVIEW:
==================================================
# AI 안전
AI 안전에 대한 초안입니다...
==================================================
이 초안을 검토해 주세요. 'approved', 'rejected'로 답하거나 수정 피드백을 제공해 주세요:
(Press Enter to skip, or type your feedback)
Your feedback: 좋아 보입니다, 승인!
✅ 콘텐츠가 승인되어 출판되었습니다!
검토자 코멘트: 좋아 보입니다, 승인!
Flow 완료. 요청된 수정: 0
```
</CodeGroup>
## 다른 데코레이터와 결합하기
`@human_feedback` 데코레이터는 다른 Flow 데코레이터와 함께 작동합니다. 가장 안쪽 데코레이터(함수에 가장 가까운)로 배치하세요:
```python Code
# 올바름: @human_feedback이 가장 안쪽(함수에 가장 가까움)
@start()
@human_feedback(message="이것을 검토해 주세요:")
def my_start_method(self):
return "content"
@listen(other_method)
@human_feedback(message="이것도 검토해 주세요:")
def my_listener(self, data):
return f"processed: {data}"
```
<Tip>
`@human_feedback`를 가장 안쪽 데코레이터(마지막/함수에 가장 가까움)로 배치하여 메서드를 직접 래핑하고 Flow 시스템에 전달하기 전에 반환 값을 캡처할 수 있도록 하세요.
</Tip>
## 모범 사례
### 1. 명확한 요청 메시지 작성
`message` 매개변수는 인간이 보는 것입니다. 실행 가능하게 만드세요:
```python Code
# ✅ 좋음 - 명확하고 실행 가능
@human_feedback(message="이 요약이 핵심 포인트를 정확하게 캡처했나요? '예'로 답하거나 무엇이 빠졌는지 설명해 주세요:")
# ❌ 나쁨 - 모호함
@human_feedback(message="이것을 검토해 주세요:")
```
### 2. 의미 있는 Outcome 선택
`emit`을 사용할 때, 인간의 응답에 자연스럽게 매핑되는 outcome을 선택하세요:
```python Code
# ✅ 좋음 - 자연어 outcome
emit=["approved", "rejected", "needs_more_detail"]
# ❌ 나쁨 - 기술적이거나 불명확
emit=["state_1", "state_2", "state_3"]
```
### 3. 항상 기본 Outcome 제공
사용자가 입력 없이 Enter를 누르는 경우를 처리하기 위해 `default_outcome`을 사용하세요:
```python Code
@human_feedback(
message="승인하시겠습니까? (수정 요청하려면 Enter 누르세요)",
emit=["approved", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision", # 안전한 기본값
)
```
### 4. 감사 추적을 위한 피드백 히스토리 사용
감사 로그를 생성하기 위해 `human_feedback_history`에 접근하세요:
```python Code
@listen(final_step)
def create_audit_log(self):
log = []
for fb in self.human_feedback_history:
log.append({
"step": fb.method_name,
"outcome": fb.outcome,
"feedback": fb.feedback,
"timestamp": fb.timestamp.isoformat(),
})
return log
```
### 5. 라우팅된 피드백과 라우팅되지 않은 피드백 모두 처리
Flow를 설계할 때, 라우팅이 필요한지 고려하세요:
| 시나리오 | 사용 |
|----------|------|
| 간단한 검토, 피드백 텍스트만 필요 | `emit` 없음 |
| 응답에 따라 다른 경로로 분기 필요 | `emit` 사용 |
| 승인/거부/수정이 있는 승인 게이트 | `emit` 사용 |
| 로깅만을 위한 코멘트 수집 | `emit` 없음 |
## 비동기 인간 피드백 (논블로킹)
기본적으로 `@human_feedback`은 콘솔 입력을 기다리며 실행을 차단합니다. 프로덕션 애플리케이션에서는 Slack, 이메일, 웹훅 또는 API와 같은 외부 시스템과 통합되는 **비동기/논블로킹** 피드백이 필요할 수 있습니다.
### Provider 추상화
커스텀 피드백 수집 전략을 지정하려면 `provider` 매개변수를 사용하세요:
```python Code
from crewai.flow import Flow, start, human_feedback, HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
class WebhookProvider(HumanFeedbackProvider):
"""웹훅 콜백을 기다리며 Flow를 일시 중지하는 Provider."""
def __init__(self, webhook_url: str):
self.webhook_url = webhook_url
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# 외부 시스템에 알림 (예: Slack 메시지 전송, 티켓 생성)
self.send_notification(context)
# 실행 일시 중지 - 프레임워크가 자동으로 영속성 처리
raise HumanFeedbackPending(
context=context,
callback_info={"webhook_url": f"{self.webhook_url}/{context.flow_id}"}
)
class ReviewFlow(Flow):
@start()
@human_feedback(
message="이 콘텐츠를 검토해 주세요:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=WebhookProvider("https://myapp.com/api"),
)
def generate_content(self):
return "AI가 생성한 콘텐츠..."
@listen("approved")
def publish(self, result):
return "출판됨!"
```
<Tip>
Flow 프레임워크는 `HumanFeedbackPending`이 발생하면 **자동으로 상태를 영속화**합니다. Provider는 외부 시스템에 알리고 예외를 발생시키기만 하면 됩니다—수동 영속성 호출이 필요하지 않습니다.
</Tip>
### 일시 중지된 Flow 처리
비동기 provider를 사용하면 `kickoff()`는 예외를 발생시키는 대신 `HumanFeedbackPending` 객체를 반환합니다:
```python Code
flow = ReviewFlow()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
# Flow가 일시 중지됨, 상태가 자동으로 영속화됨
print(f"피드백 대기 중: {result.callback_info['webhook_url']}")
print(f"Flow ID: {result.context.flow_id}")
else:
# 정상 완료
print(f"Flow 완료: {result}")
```
### 일시 중지된 Flow 재개
피드백이 도착하면 (예: 웹훅을 통해) Flow를 재개합니다:
```python Code
# 동기 핸들러:
def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = flow.resume(feedback)
return result
# 비동기 핸들러 (FastAPI, aiohttp 등):
async def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = await flow.resume_async(feedback)
return result
```
### 주요 타입
| 타입 | 설명 |
|------|------|
| `HumanFeedbackProvider` | 커스텀 피드백 provider를 위한 프로토콜 |
| `PendingFeedbackContext` | 일시 중지된 Flow를 재개하는 데 필요한 모든 정보 포함 |
| `HumanFeedbackPending` | Flow가 피드백을 위해 일시 중지되면 `kickoff()`에서 반환됨 |
| `ConsoleProvider` | 기본 블로킹 콘솔 입력 provider |
### PendingFeedbackContext
컨텍스트는 재개에 필요한 모든 것을 포함합니다:
```python Code
@dataclass
class PendingFeedbackContext:
flow_id: str # 이 Flow 실행의 고유 식별자
flow_class: str # 정규화된 클래스 이름
method_name: str # 피드백을 트리거한 메서드
method_output: Any # 인간에게 표시된 출력
message: str # 요청 메시지
emit: list[str] | None # 라우팅을 위한 가능한 outcome
default_outcome: str | None
metadata: dict # 커스텀 메타데이터
llm: str | None # outcome 매핑을 위한 LLM
requested_at: datetime
```
### 완전한 비동기 Flow 예제
```python Code
from crewai.flow import (
Flow, start, listen, human_feedback,
HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
)
class SlackNotificationProvider(HumanFeedbackProvider):
"""Slack 알림을 보내고 비동기 피드백을 위해 일시 중지하는 Provider."""
def __init__(self, channel: str):
self.channel = channel
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# Slack 알림 전송 (직접 구현)
slack_thread_id = self.post_to_slack(
channel=self.channel,
message=f"검토 필요:\n\n{context.method_output}\n\n{context.message}",
)
# 실행 일시 중지 - 프레임워크가 자동으로 영속성 처리
raise HumanFeedbackPending(
context=context,
callback_info={
"slack_channel": self.channel,
"thread_id": slack_thread_id,
}
)
class ContentPipeline(Flow):
@start()
@human_feedback(
message="이 콘텐츠의 출판을 승인하시겠습니까?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
provider=SlackNotificationProvider("#content-reviews"),
)
def generate_content(self):
return "AI가 생성한 블로그 게시물 콘텐츠..."
@listen("approved")
def publish(self, result):
print(f"출판 중! 검토자 의견: {result.feedback}")
return {"status": "published"}
@listen("rejected")
def archive(self, result):
print(f"보관됨. 이유: {result.feedback}")
return {"status": "archived"}
@listen("needs_revision")
def queue_revision(self, result):
print(f"수정 대기열에 추가됨: {result.feedback}")
return {"status": "revision_needed"}
# Flow 시작 (Slack 응답을 기다리며 일시 중지)
def start_content_pipeline():
flow = ContentPipeline()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
return {"status": "pending", "flow_id": result.context.flow_id}
return result
# Slack 웹훅이 실행될 때 재개 (동기 핸들러)
def on_slack_feedback(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = flow.resume(slack_message)
return result
# 핸들러가 비동기인 경우 (FastAPI, aiohttp, Slack Bolt 비동기 등)
async def on_slack_feedback_async(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = await flow.resume_async(slack_message)
return result
```
<Warning>
비동기 웹 프레임워크(FastAPI, aiohttp, Slack Bolt 비동기 모드)를 사용하는 경우 `flow.resume()` 대신 `await flow.resume_async()`를 사용하세요. 실행 중인 이벤트 루프 내에서 `resume()`을 호출하면 `RuntimeError`가 발생합니다.
</Warning>
### 비동기 피드백 모범 사례
1. **반환 타입 확인**: `kickoff()`는 일시 중지되면 `HumanFeedbackPending`을 반환합니다—try/except가 필요하지 않습니다
2. **올바른 resume 메서드 사용**: 동기 코드에서는 `resume()`, 비동기 코드에서는 `await resume_async()` 사용
3. **콜백 정보 저장**: `callback_info`를 사용하여 웹훅 URL, 티켓 ID 등을 저장
4. **멱등성 구현**: 안전을 위해 resume 핸들러는 멱등해야 합니다
5. **자동 영속성**: `HumanFeedbackPending`이 발생하면 상태가 자동으로 저장되며 기본적으로 `SQLiteFlowPersistence` 사용
6. **커스텀 영속성**: 필요한 경우 `from_pending()`에 커스텀 영속성 인스턴스 전달
## 관련 문서
- [Flow 개요](/ko/concepts/flows) - CrewAI Flow에 대해 알아보기
- [Flow 상태 관리](/ko/guides/flows/mastering-flow-state) - Flow에서 상태 관리하기
- [Flow 영속성](/ko/concepts/flows#persistence) - Flow 상태 영속화
- [@router를 사용한 라우팅](/ko/concepts/flows#router) - 조건부 라우팅에 대해 더 알아보기
- [실행 시 인간 입력](/ko/learn/human-input-on-execution) - 태스크 수준 인간 입력

View File

@@ -307,6 +307,55 @@ Os métodos `third_method` e `fourth_method` escutam a saída do `second_method`
Ao executar esse Flow, a saída será diferente dependendo do valor booleano aleatório gerado pelo `start_method`.
### Human in the Loop (feedback humano)
O decorador `@human_feedback` permite fluxos de trabalho human-in-the-loop, pausando a execução do flow para coletar feedback de um humano. Isso é útil para portões de aprovação, revisão de qualidade e pontos de decisão que requerem julgamento humano.
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Você aprova este conteúdo?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def generate_content(self):
return "Conteúdo para revisão..."
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"Aprovado! Feedback: {result.feedback}")
@listen("rejected")
def on_rejection(self, result: HumanFeedbackResult):
print(f"Rejeitado. Motivo: {result.feedback}")
```
Quando `emit` é especificado, o feedback livre do humano é interpretado por um LLM e mapeado para um dos outcomes especificados, que então dispara o decorador `@listen` correspondente.
Você também pode usar `@human_feedback` sem roteamento para simplesmente coletar feedback:
```python Code
@start()
@human_feedback(message="Algum comentário sobre esta saída?")
def my_method(self):
return "Saída para revisão"
@listen(my_method)
def next_step(self, result: HumanFeedbackResult):
# Acesse o feedback via result.feedback
# Acesse a saída original via result.output
pass
```
Acesse todo o feedback coletado durante um flow via `self.last_human_feedback` (mais recente) ou `self.human_feedback_history` (todo o feedback em uma lista).
Para um guia completo sobre feedback humano em flows, incluindo feedback assíncrono/não-bloqueante com providers customizados (Slack, webhooks, etc.), veja [Feedback Humano em Flows](/pt-BR/learn/human-feedback-in-flows).
## Adicionando Agentes aos Flows
Os agentes podem ser integrados facilmente aos seus flows, oferecendo uma alternativa leve às crews completas quando você precisar executar tarefas simples e focadas. Veja um exemplo de como utilizar um agente em um flow para realizar uma pesquisa de mercado:

View File

@@ -62,13 +62,13 @@ Teste sua integração de trigger do Gmail localmente usando a CLI da CrewAI:
crewai triggers list
# Simule um trigger do Gmail com payload realista
crewai triggers run gmail/new_email
crewai triggers run gmail/new_email_received
```
O comando `crewai triggers run` executará sua crew com um payload completo do Gmail, permitindo que você teste sua lógica de parsing antes do deployment.
<Warning>
Use `crewai triggers run gmail/new_email` (não `crewai run`) para simular execução de trigger durante o desenvolvimento. Após o deployment, sua crew receberá automaticamente o payload do trigger.
Use `crewai triggers run gmail/new_email_received` (não `crewai run`) para simular execução de trigger durante o desenvolvimento. Após o deployment, sua crew receberá automaticamente o payload do trigger.
</Warning>
## Monitoring Executions
@@ -83,6 +83,6 @@ Track history and performance of triggered runs:
- Ensure Gmail is connected in Tools & Integrations
- Verify the Gmail Trigger is enabled on the Triggers tab
- Teste localmente com `crewai triggers run gmail/new_email` para ver a estrutura exata do payload
- Teste localmente com `crewai triggers run gmail/new_email_received` para ver a estrutura exata do payload
- Check the execution logs and confirm the payload is passed as `crewai_trigger_payload`
- Lembre-se: use `crewai triggers run` (não `crewai run`) para simular execução de trigger

View File

@@ -0,0 +1,581 @@
---
title: Feedback Humano em Flows
description: Aprenda como integrar feedback humano diretamente nos seus CrewAI Flows usando o decorador @human_feedback
icon: user-check
mode: "wide"
---
## Visão Geral
O decorador `@human_feedback` permite fluxos de trabalho human-in-the-loop (HITL) diretamente nos CrewAI Flows. Ele permite pausar a execução do flow, apresentar a saída para um humano revisar, coletar seu feedback e, opcionalmente, rotear para diferentes listeners com base no resultado do feedback.
Isso é particularmente valioso para:
- **Garantia de qualidade**: Revisar conteúdo gerado por IA antes de ser usado downstream
- **Portões de decisão**: Deixar humanos tomarem decisões críticas em fluxos automatizados
- **Fluxos de aprovação**: Implementar padrões de aprovar/rejeitar/revisar
- **Refinamento interativo**: Coletar feedback para melhorar saídas iterativamente
```mermaid
flowchart LR
A[Método do Flow] --> B[Saída Gerada]
B --> C[Humano Revisa]
C --> D{Feedback}
D -->|emit especificado| E[LLM Mapeia para Outcome]
D -->|sem emit| F[HumanFeedbackResult]
E --> G["@listen('approved')"]
E --> H["@listen('rejected')"]
F --> I[Próximo Listener]
```
## Início Rápido
Aqui está a maneira mais simples de adicionar feedback humano a um flow:
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback
class SimpleReviewFlow(Flow):
@start()
@human_feedback(message="Por favor, revise este conteúdo:")
def generate_content(self):
return "Este é um conteúdo gerado por IA que precisa de revisão."
@listen(generate_content)
def process_feedback(self, result):
print(f"Conteúdo: {result.output}")
print(f"Humano disse: {result.feedback}")
flow = SimpleReviewFlow()
flow.kickoff()
```
Quando este flow é executado, ele irá:
1. Executar `generate_content` e retornar a string
2. Exibir a saída para o usuário com a mensagem de solicitação
3. Aguardar o usuário digitar o feedback (ou pressionar Enter para pular)
4. Passar um objeto `HumanFeedbackResult` para `process_feedback`
## O Decorador @human_feedback
### Parâmetros
| Parâmetro | Tipo | Obrigatório | Descrição |
|-----------|------|-------------|-----------|
| `message` | `str` | Sim | A mensagem mostrada ao humano junto com a saída do método |
| `emit` | `Sequence[str]` | Não | Lista de possíveis outcomes. O feedback é mapeado para um destes, que dispara decoradores `@listen` |
| `llm` | `str \| BaseLLM` | Quando `emit` especificado | LLM usado para interpretar o feedback e mapear para um outcome |
| `default_outcome` | `str` | Não | Outcome a usar se nenhum feedback for fornecido. Deve estar em `emit` |
| `metadata` | `dict` | Não | Dados adicionais para integrações enterprise |
| `provider` | `HumanFeedbackProvider` | Não | Provider customizado para feedback assíncrono/não-bloqueante. Veja [Feedback Humano Assíncrono](#feedback-humano-assíncrono-não-bloqueante) |
### Uso Básico (Sem Roteamento)
Quando você não especifica `emit`, o decorador simplesmente coleta o feedback e passa um `HumanFeedbackResult` para o próximo listener:
```python Code
@start()
@human_feedback(message="O que você acha desta análise?")
def analyze_data(self):
return "Resultados da análise: Receita aumentou 15%, custos diminuíram 8%"
@listen(analyze_data)
def handle_feedback(self, result):
# result é um HumanFeedbackResult
print(f"Análise: {result.output}")
print(f"Feedback: {result.feedback}")
```
### Roteamento com emit
Quando você especifica `emit`, o decorador se torna um roteador. O feedback livre do humano é interpretado por um LLM e mapeado para um dos outcomes especificados:
```python Code
@start()
@human_feedback(
message="Você aprova este conteúdo para publicação?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_content(self):
return "Rascunho do post do blog aqui..."
@listen("approved")
def publish(self, result):
print(f"Publicando! Usuário disse: {result.feedback}")
@listen("rejected")
def discard(self, result):
print(f"Descartando. Motivo: {result.feedback}")
@listen("needs_revision")
def revise(self, result):
print(f"Revisando baseado em: {result.feedback}")
```
<Tip>
O LLM usa saídas estruturadas (function calling) quando disponível para garantir que a resposta seja um dos seus outcomes especificados. Isso torna o roteamento confiável e previsível.
</Tip>
## HumanFeedbackResult
O dataclass `HumanFeedbackResult` contém todas as informações sobre uma interação de feedback humano:
```python Code
from crewai.flow.human_feedback import HumanFeedbackResult
@dataclass
class HumanFeedbackResult:
output: Any # A saída original do método mostrada ao humano
feedback: str # O texto bruto do feedback do humano
outcome: str | None # O outcome mapeado (se emit foi especificado)
timestamp: datetime # Quando o feedback foi recebido
method_name: str # Nome do método decorado
metadata: dict # Qualquer metadata passado ao decorador
```
### Acessando em Listeners
Quando um listener é disparado por um método `@human_feedback` com `emit`, ele recebe o `HumanFeedbackResult`:
```python Code
@listen("approved")
def on_approval(self, result: HumanFeedbackResult):
print(f"Saída original: {result.output}")
print(f"Feedback do usuário: {result.feedback}")
print(f"Outcome: {result.outcome}") # "approved"
print(f"Recebido em: {result.timestamp}")
```
## Acessando o Histórico de Feedback
A classe `Flow` fornece dois atributos para acessar o feedback humano:
### last_human_feedback
Retorna o `HumanFeedbackResult` mais recente:
```python Code
@listen(some_method)
def check_feedback(self):
if self.last_human_feedback:
print(f"Último feedback: {self.last_human_feedback.feedback}")
```
### human_feedback_history
Uma lista de todos os objetos `HumanFeedbackResult` coletados durante o flow:
```python Code
@listen(final_step)
def summarize(self):
print(f"Total de feedbacks coletados: {len(self.human_feedback_history)}")
for i, fb in enumerate(self.human_feedback_history):
print(f"{i+1}. {fb.method_name}: {fb.outcome or 'sem roteamento'}")
```
<Warning>
Cada `HumanFeedbackResult` é adicionado a `human_feedback_history`, então múltiplos passos de feedback não sobrescrevem uns aos outros. Use esta lista para acessar todo o feedback coletado durante o flow.
</Warning>
## Exemplo Completo: Fluxo de Aprovação de Conteúdo
Aqui está um exemplo completo implementando um fluxo de revisão e aprovação de conteúdo:
<CodeGroup>
```python Code
from crewai.flow.flow import Flow, start, listen
from crewai.flow.human_feedback import human_feedback, HumanFeedbackResult
from pydantic import BaseModel
class ContentState(BaseModel):
topic: str = ""
draft: str = ""
final_content: str = ""
revision_count: int = 0
class ContentApprovalFlow(Flow[ContentState]):
"""Um flow que gera conteúdo e obtém aprovação humana."""
@start()
def get_topic(self):
self.state.topic = input("Sobre qual tópico devo escrever? ")
return self.state.topic
@listen(get_topic)
def generate_draft(self, topic):
# Em uso real, isso chamaria um LLM
self.state.draft = f"# {topic}\n\nEste é um rascunho sobre {topic}..."
return self.state.draft
@listen(generate_draft)
@human_feedback(
message="Por favor, revise este rascunho. Responda 'approved', 'rejected', ou forneça feedback de revisão:",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_draft(self, draft):
return draft
@listen("approved")
def publish_content(self, result: HumanFeedbackResult):
self.state.final_content = result.output
print("\n✅ Conteúdo aprovado e publicado!")
print(f"Comentário do revisor: {result.feedback}")
return "published"
@listen("rejected")
def handle_rejection(self, result: HumanFeedbackResult):
print("\n❌ Conteúdo rejeitado")
print(f"Motivo: {result.feedback}")
return "rejected"
@listen("needs_revision")
def revise_content(self, result: HumanFeedbackResult):
self.state.revision_count += 1
print(f"\n📝 Revisão #{self.state.revision_count} solicitada")
print(f"Feedback: {result.feedback}")
# Em um flow real, você pode voltar para generate_draft
# Para este exemplo, apenas reconhecemos
return "revision_requested"
# Executar o flow
flow = ContentApprovalFlow()
result = flow.kickoff()
print(f"\nFlow concluído. Revisões solicitadas: {flow.state.revision_count}")
```
```text Output
Sobre qual tópico devo escrever? Segurança em IA
==================================================
OUTPUT FOR REVIEW:
==================================================
# Segurança em IA
Este é um rascunho sobre Segurança em IA...
==================================================
Por favor, revise este rascunho. Responda 'approved', 'rejected', ou forneça feedback de revisão:
(Press Enter to skip, or type your feedback)
Your feedback: Parece bom, aprovado!
✅ Conteúdo aprovado e publicado!
Comentário do revisor: Parece bom, aprovado!
Flow concluído. Revisões solicitadas: 0
```
</CodeGroup>
## Combinando com Outros Decoradores
O decorador `@human_feedback` funciona com outros decoradores de flow. Coloque-o como o decorador mais interno (mais próximo da função):
```python Code
# Correto: @human_feedback é o mais interno (mais próximo da função)
@start()
@human_feedback(message="Revise isto:")
def my_start_method(self):
return "content"
@listen(other_method)
@human_feedback(message="Revise isto também:")
def my_listener(self, data):
return f"processed: {data}"
```
<Tip>
Coloque `@human_feedback` como o decorador mais interno (último/mais próximo da função) para que ele envolva o método diretamente e possa capturar o valor de retorno antes de passar para o sistema de flow.
</Tip>
## Melhores Práticas
### 1. Escreva Mensagens de Solicitação Claras
O parâmetro `message` é o que o humano vê. Torne-o acionável:
```python Code
# ✅ Bom - claro e acionável
@human_feedback(message="Este resumo captura com precisão os pontos-chave? Responda 'sim' ou explique o que está faltando:")
# ❌ Ruim - vago
@human_feedback(message="Revise isto:")
```
### 2. Escolha Outcomes Significativos
Ao usar `emit`, escolha outcomes que mapeiem naturalmente para respostas humanas:
```python Code
# ✅ Bom - outcomes em linguagem natural
emit=["approved", "rejected", "needs_more_detail"]
# ❌ Ruim - técnico ou pouco claro
emit=["state_1", "state_2", "state_3"]
```
### 3. Sempre Forneça um Outcome Padrão
Use `default_outcome` para lidar com casos onde usuários pressionam Enter sem digitar:
```python Code
@human_feedback(
message="Aprovar? (pressione Enter para solicitar revisão)",
emit=["approved", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision", # Padrão seguro
)
```
### 4. Use o Histórico de Feedback para Trilhas de Auditoria
Acesse `human_feedback_history` para criar logs de auditoria:
```python Code
@listen(final_step)
def create_audit_log(self):
log = []
for fb in self.human_feedback_history:
log.append({
"step": fb.method_name,
"outcome": fb.outcome,
"feedback": fb.feedback,
"timestamp": fb.timestamp.isoformat(),
})
return log
```
### 5. Trate Feedback Roteado e Não Roteado
Ao projetar flows, considere se você precisa de roteamento:
| Cenário | Use |
|---------|-----|
| Revisão simples, só precisa do texto do feedback | Sem `emit` |
| Precisa ramificar para caminhos diferentes baseado na resposta | Use `emit` |
| Portões de aprovação com aprovar/rejeitar/revisar | Use `emit` |
| Coletando comentários apenas para logging | Sem `emit` |
## Feedback Humano Assíncrono (Não-Bloqueante - Human in the loop)
Por padrão, `@human_feedback` bloqueia a execução aguardando entrada no console. Para aplicações de produção, você pode precisar de feedback **assíncrono/não-bloqueante** que se integre com sistemas externos como Slack, email, webhooks ou APIs.
### A Abstração de Provider
Use o parâmetro `provider` para especificar uma estratégia customizada de coleta de feedback:
```python Code
from crewai.flow import Flow, start, human_feedback, HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
class WebhookProvider(HumanFeedbackProvider):
"""Provider que pausa o flow e aguarda callback de webhook."""
def __init__(self, webhook_url: str):
self.webhook_url = webhook_url
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# Notifica sistema externo (ex: envia mensagem Slack, cria ticket)
self.send_notification(context)
# Pausa execução - framework cuida da persistência automaticamente
raise HumanFeedbackPending(
context=context,
callback_info={"webhook_url": f"{self.webhook_url}/{context.flow_id}"}
)
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Revise este conteúdo:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=WebhookProvider("https://myapp.com/api"),
)
def generate_content(self):
return "Conteúdo gerado por IA..."
@listen("approved")
def publish(self, result):
return "Publicado!"
```
<Tip>
O framework de flow **persiste automaticamente o estado** quando `HumanFeedbackPending` é lançado. Seu provider só precisa notificar o sistema externo e lançar a exceção—não são necessárias chamadas manuais de persistência.
</Tip>
### Tratando Flows Pausados
Ao usar um provider assíncrono, `kickoff()` retorna um objeto `HumanFeedbackPending` em vez de lançar uma exceção:
```python Code
flow = ReviewFlow()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
# Flow está pausado, estado é automaticamente persistido
print(f"Aguardando feedback em: {result.callback_info['webhook_url']}")
print(f"Flow ID: {result.context.flow_id}")
else:
# Conclusão normal
print(f"Flow concluído: {result}")
```
### Retomando um Flow Pausado
Quando o feedback chega (ex: via webhook), retome o flow:
```python Code
# Handler síncrono:
def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = flow.resume(feedback)
return result
# Handler assíncrono (FastAPI, aiohttp, etc.):
async def handle_feedback_webhook(flow_id: str, feedback: str):
flow = ReviewFlow.from_pending(flow_id)
result = await flow.resume_async(feedback)
return result
```
### Tipos Principais
| Tipo | Descrição |
|------|-----------|
| `HumanFeedbackProvider` | Protocolo para providers de feedback customizados |
| `PendingFeedbackContext` | Contém todas as informações necessárias para retomar um flow pausado |
| `HumanFeedbackPending` | Retornado por `kickoff()` quando o flow está pausado para feedback |
| `ConsoleProvider` | Provider padrão de entrada bloqueante no console |
### PendingFeedbackContext
O contexto contém tudo necessário para retomar:
```python Code
@dataclass
class PendingFeedbackContext:
flow_id: str # Identificador único desta execução de flow
flow_class: str # Nome qualificado completo da classe
method_name: str # Método que disparou o feedback
method_output: Any # Saída mostrada ao humano
message: str # A mensagem de solicitação
emit: list[str] | None # Outcomes possíveis para roteamento
default_outcome: str | None
metadata: dict # Metadata customizado
llm: str | None # LLM para mapeamento de outcome
requested_at: datetime
```
### Exemplo Completo de Flow Assíncrono
```python Code
from crewai.flow import (
Flow, start, listen, human_feedback,
HumanFeedbackProvider, HumanFeedbackPending, PendingFeedbackContext
)
class SlackNotificationProvider(HumanFeedbackProvider):
"""Provider que envia notificações Slack e pausa para feedback assíncrono."""
def __init__(self, channel: str):
self.channel = channel
def request_feedback(self, context: PendingFeedbackContext, flow: Flow) -> str:
# Envia notificação Slack (implemente você mesmo)
slack_thread_id = self.post_to_slack(
channel=self.channel,
message=f"Revisão necessária:\n\n{context.method_output}\n\n{context.message}",
)
# Pausa execução - framework cuida da persistência automaticamente
raise HumanFeedbackPending(
context=context,
callback_info={
"slack_channel": self.channel,
"thread_id": slack_thread_id,
}
)
class ContentPipeline(Flow):
@start()
@human_feedback(
message="Aprova este conteúdo para publicação?",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
provider=SlackNotificationProvider("#content-reviews"),
)
def generate_content(self):
return "Conteúdo de blog post gerado por IA..."
@listen("approved")
def publish(self, result):
print(f"Publicando! Revisor disse: {result.feedback}")
return {"status": "published"}
@listen("rejected")
def archive(self, result):
print(f"Arquivado. Motivo: {result.feedback}")
return {"status": "archived"}
@listen("needs_revision")
def queue_revision(self, result):
print(f"Na fila para revisão: {result.feedback}")
return {"status": "revision_needed"}
# Iniciando o flow (vai pausar e aguardar resposta do Slack)
def start_content_pipeline():
flow = ContentPipeline()
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
return {"status": "pending", "flow_id": result.context.flow_id}
return result
# Retomando quando webhook do Slack dispara (handler síncrono)
def on_slack_feedback(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = flow.resume(slack_message)
return result
# Se seu handler é assíncrono (FastAPI, aiohttp, Slack Bolt async, etc.)
async def on_slack_feedback_async(flow_id: str, slack_message: str):
flow = ContentPipeline.from_pending(flow_id)
result = await flow.resume_async(slack_message)
return result
```
<Warning>
Se você está usando um framework web assíncrono (FastAPI, aiohttp, Slack Bolt modo async), use `await flow.resume_async()` em vez de `flow.resume()`. Chamar `resume()` de dentro de um event loop em execução vai lançar um `RuntimeError`.
</Warning>
### Melhores Práticas para Feedback Assíncrono
1. **Verifique o tipo de retorno**: `kickoff()` retorna `HumanFeedbackPending` quando pausado—não precisa de try/except
2. **Use o método resume correto**: Use `resume()` em código síncrono, `await resume_async()` em código assíncrono
3. **Armazene informações de callback**: Use `callback_info` para armazenar URLs de webhook, IDs de tickets, etc.
4. **Implemente idempotência**: Seu handler de resume deve ser idempotente por segurança
5. **Persistência automática**: O estado é automaticamente salvo quando `HumanFeedbackPending` é lançado e usa `SQLiteFlowPersistence` por padrão
6. **Persistência customizada**: Passe uma instância de persistência customizada para `from_pending()` se necessário
## Documentação Relacionada
- [Visão Geral de Flows](/pt-BR/concepts/flows) - Aprenda sobre CrewAI Flows
- [Gerenciamento de Estado em Flows](/pt-BR/guides/flows/mastering-flow-state) - Gerenciando estado em flows
- [Persistência de Flows](/pt-BR/concepts/flows#persistence) - Persistindo estado de flows
- [Roteamento com @router](/pt-BR/concepts/flows#router) - Mais sobre roteamento condicional
- [Input Humano na Execução](/pt-BR/learn/human-input-on-execution) - Input humano no nível de task

View File

@@ -1,7 +1,7 @@
from __future__ import annotations
import asyncio
from collections.abc import Sequence
from collections.abc import Callable, Sequence
import shutil
import subprocess
import time
@@ -44,6 +44,7 @@ from crewai.events.types.memory_events import (
MemoryRetrievalCompletedEvent,
MemoryRetrievalStartedEvent,
)
from crewai.experimental.crew_agent_executor_flow import CrewAgentExecutorFlow
from crewai.knowledge.knowledge import Knowledge
from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSource
from crewai.lite_agent import LiteAgent
@@ -105,7 +106,7 @@ class Agent(BaseAgent):
The agent can also have memory, can operate in verbose mode, and can delegate tasks to other agents.
Attributes:
agent_executor: An instance of the CrewAgentExecutor class.
agent_executor: An instance of the CrewAgentExecutor or CrewAgentExecutorFlow class.
role: The role of the agent.
goal: The objective of the agent.
backstory: The backstory of the agent.
@@ -221,6 +222,10 @@ class Agent(BaseAgent):
default=None,
description="A2A (Agent-to-Agent) configuration for delegating tasks to remote agents. Can be a single A2AConfig or a dict mapping agent IDs to configs.",
)
executor_class: type[CrewAgentExecutor] | type[CrewAgentExecutorFlow] = Field(
default=CrewAgentExecutor,
description="Class to use for the agent executor. Defaults to CrewAgentExecutor, can optionally use CrewAgentExecutorFlow.",
)
@model_validator(mode="before")
def validate_from_repository(cls, v: Any) -> dict[str, Any] | None | Any: # noqa: N805
@@ -439,12 +444,6 @@ class Agent(BaseAgent):
def _execute_with_timeout(self, task_prompt: str, task: Task, timeout: int) -> Any:
"""Execute a task with a timeout.
This method uses cooperative cancellation to ensure clean thread cleanup.
When a timeout occurs:
1. The executor's deadline is set, causing the worker to check and exit
2. The executor is shut down with wait=False to return control promptly
3. The worker thread will exit cleanly when it checks the deadline
Args:
task_prompt: The prompt to send to the agent.
task: The task being executed.
@@ -459,11 +458,7 @@ class Agent(BaseAgent):
"""
import concurrent.futures
if self.agent_executor:
self.agent_executor.set_execution_deadline(timeout)
executor = concurrent.futures.ThreadPoolExecutor(thread_name_prefix="crewai_task")
try:
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(
self._execute_without_timeout, task_prompt=task_prompt, task=task
)
@@ -478,10 +473,6 @@ class Agent(BaseAgent):
except Exception as e:
future.cancel()
raise RuntimeError(f"Task execution failed: {e!s}") from e
finally:
if self.agent_executor:
self.agent_executor.clear_execution_deadline()
executor.shutdown(wait=False)
def _execute_without_timeout(self, task_prompt: str, task: Task) -> Any:
"""Execute a task without a timeout.
@@ -661,12 +652,6 @@ class Agent(BaseAgent):
) -> Any:
"""Execute a task with a timeout asynchronously.
This method uses cooperative cancellation to ensure clean task cleanup.
When a timeout occurs:
1. The executor's deadline is set, causing the worker to check and exit
2. asyncio.wait_for cancels the coroutine
3. The worker will exit cleanly when it checks the deadline
Args:
task_prompt: The prompt to send to the agent.
task: The task being executed.
@@ -679,9 +664,6 @@ class Agent(BaseAgent):
TimeoutError: If execution exceeds the timeout.
RuntimeError: If execution fails for other reasons.
"""
if self.agent_executor:
self.agent_executor.set_execution_deadline(timeout)
try:
return await asyncio.wait_for(
self._aexecute_without_timeout(task_prompt, task),
@@ -692,9 +674,6 @@ class Agent(BaseAgent):
f"Task '{task.description}' execution timed out after {timeout} seconds. "
"Consider increasing max_execution_time or optimizing the task."
) from e
finally:
if self.agent_executor:
self.agent_executor.clear_execution_deadline()
async def _aexecute_without_timeout(self, task_prompt: str, task: Task) -> Any:
"""Execute a task without a timeout asynchronously.
@@ -747,29 +726,83 @@ class Agent(BaseAgent):
self.response_template.split("{{ .Response }}")[1].strip()
)
self.agent_executor = CrewAgentExecutor(
llm=self.llm, # type: ignore[arg-type]
task=task, # type: ignore[arg-type]
agent=self,
crew=self.crew,
tools=parsed_tools,
prompt=prompt,
original_tools=raw_tools,
stop_words=stop_words,
max_iter=self.max_iter,
tools_handler=self.tools_handler,
tools_names=get_tool_names(parsed_tools),
tools_description=render_text_description_and_args(parsed_tools),
step_callback=self.step_callback,
function_calling_llm=self.function_calling_llm,
respect_context_window=self.respect_context_window,
request_within_rpm_limit=(
self._rpm_controller.check_or_wait if self._rpm_controller else None
),
callbacks=[TokenCalcHandler(self._token_process)],
response_model=task.response_model if task else None,
rpm_limit_fn = (
self._rpm_controller.check_or_wait if self._rpm_controller else None
)
if self.agent_executor is not None:
self._update_executor_parameters(
task=task,
tools=parsed_tools,
raw_tools=raw_tools,
prompt=prompt,
stop_words=stop_words,
rpm_limit_fn=rpm_limit_fn,
)
else:
self.agent_executor = self.executor_class(
llm=cast(BaseLLM, self.llm),
task=task,
i18n=self.i18n,
agent=self,
crew=self.crew,
tools=parsed_tools,
prompt=prompt,
original_tools=raw_tools,
stop_words=stop_words,
max_iter=self.max_iter,
tools_handler=self.tools_handler,
tools_names=get_tool_names(parsed_tools),
tools_description=render_text_description_and_args(parsed_tools),
step_callback=self.step_callback,
function_calling_llm=self.function_calling_llm,
respect_context_window=self.respect_context_window,
request_within_rpm_limit=rpm_limit_fn,
callbacks=[TokenCalcHandler(self._token_process)],
response_model=task.response_model if task else None,
)
def _update_executor_parameters(
self,
task: Task | None,
tools: list,
raw_tools: list[BaseTool],
prompt: dict,
stop_words: list[str],
rpm_limit_fn: Callable | None,
) -> None:
"""Update executor parameters without recreating instance.
Args:
task: Task to execute.
tools: Parsed tools.
raw_tools: Original tools.
prompt: Generated prompt.
stop_words: Stop words list.
rpm_limit_fn: RPM limit callback function.
"""
self.agent_executor.task = task
self.agent_executor.tools = tools
self.agent_executor.original_tools = raw_tools
self.agent_executor.prompt = prompt
self.agent_executor.stop = stop_words
self.agent_executor.tools_names = get_tool_names(tools)
self.agent_executor.tools_description = render_text_description_and_args(tools)
self.agent_executor.response_model = task.response_model if task else None
self.agent_executor.tools_handler = self.tools_handler
self.agent_executor.request_within_rpm_limit = rpm_limit_fn
if self.agent_executor.llm:
existing_stop = getattr(self.agent_executor.llm, "stop", [])
self.agent_executor.llm.stop = list(
set(
existing_stop + stop_words
if isinstance(existing_stop, list)
else stop_words
)
)
def get_delegation_tools(self, agents: list[BaseAgent]) -> list[BaseTool]:
agent_tools = AgentTools(agents=agents)
return agent_tools.tools()

View File

@@ -457,7 +457,6 @@ class BaseAgent(BaseModel, ABC, metaclass=AgentMeta):
if self.cache:
self.cache_handler = cache_handler
self.tools_handler.cache = cache_handler
self.create_agent_executor()
def set_rpm_controller(self, rpm_controller: RPMController) -> None:
"""Set the rpm controller for the agent.
@@ -467,7 +466,6 @@ class BaseAgent(BaseModel, ABC, metaclass=AgentMeta):
"""
if not self._rpm_controller:
self._rpm_controller = rpm_controller
self.create_agent_executor()
def set_knowledge(self, crew_embedder: EmbedderConfig | None = None) -> None:
pass

View File

@@ -91,6 +91,7 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
request_within_rpm_limit: Callable[[], bool] | None = None,
callbacks: list[Any] | None = None,
response_model: type[BaseModel] | None = None,
i18n: I18N | None = None,
) -> None:
"""Initialize executor.
@@ -114,7 +115,7 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
callbacks: Optional callbacks list.
response_model: Optional Pydantic model for structured outputs.
"""
self._i18n: I18N = get_i18n()
self._i18n: I18N = i18n or get_i18n()
self.llm = llm
self.task = task
self.agent = agent
@@ -138,7 +139,6 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
self.messages: list[LLMMessage] = []
self.iterations = 0
self.log_error_after = 3
self._execution_deadline: float | None = None
self.before_llm_call_hooks: list[Callable[..., Any]] = []
self.after_llm_call_hooks: list[Callable[..., Any]] = []
self.before_llm_call_hooks.extend(get_before_llm_call_hooks())
@@ -163,36 +163,6 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
"""
return self.llm.supports_stop_words() if self.llm else False
def set_execution_deadline(self, timeout_seconds: int | float) -> None:
"""Set the execution deadline for cooperative timeout.
Args:
timeout_seconds: Maximum execution time in seconds.
"""
import time
self._execution_deadline = time.monotonic() + timeout_seconds
def clear_execution_deadline(self) -> None:
"""Clear the execution deadline."""
self._execution_deadline = None
def _check_execution_deadline(self) -> None:
"""Check if the execution deadline has been exceeded.
Raises:
TimeoutError: If the deadline has been exceeded.
"""
import time
if self._execution_deadline is not None:
if time.monotonic() >= self._execution_deadline:
task_desc = self.task.description if self.task else "Unknown task"
raise TimeoutError(
f"Task '{task_desc}' execution timed out. "
"Consider increasing max_execution_time or optimizing the task."
)
def invoke(self, inputs: dict[str, Any]) -> dict[str, Any]:
"""Execute the agent with given inputs.
@@ -248,8 +218,6 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
formatted_answer = None
while not isinstance(formatted_answer, AgentFinish):
try:
self._check_execution_deadline()
if has_reached_max_iterations(self.iterations, self.max_iter):
formatted_answer = handle_max_iterations_exceeded(
formatted_answer,
@@ -404,8 +372,6 @@ class CrewAgentExecutor(CrewAgentExecutorMixin):
formatted_answer = None
while not isinstance(formatted_answer, AgentFinish):
try:
self._check_execution_deadline()
if has_reached_max_iterations(self.iterations, self.max_iter):
formatted_answer = handle_max_iterations_exceeded(
formatted_answer,

View File

@@ -38,9 +38,11 @@ from crewai.events.types.crew_events import (
from crewai.events.types.flow_events import (
FlowCreatedEvent,
FlowFinishedEvent,
FlowPausedEvent,
FlowStartedEvent,
MethodExecutionFailedEvent,
MethodExecutionFinishedEvent,
MethodExecutionPausedEvent,
MethodExecutionStartedEvent,
)
from crewai.events.types.knowledge_events import (
@@ -363,6 +365,28 @@ class EventListener(BaseEventListener):
)
self.method_branches[event.method_name] = updated_branch
@crewai_event_bus.on(MethodExecutionPausedEvent)
def on_method_execution_paused(
_: Any, event: MethodExecutionPausedEvent
) -> None:
method_branch = self.method_branches.get(event.method_name)
updated_branch = self.formatter.update_method_status(
method_branch,
self.formatter.current_flow_tree,
event.method_name,
"paused",
)
self.method_branches[event.method_name] = updated_branch
@crewai_event_bus.on(FlowPausedEvent)
def on_flow_paused(_: Any, event: FlowPausedEvent) -> None:
self.formatter.update_flow_status(
self.formatter.current_flow_tree,
event.flow_name,
event.flow_id,
"paused",
)
# ----------- TOOL USAGE EVENTS -----------
@crewai_event_bus.on(ToolUsageStartedEvent)

View File

@@ -58,6 +58,29 @@ class MethodExecutionFailedEvent(FlowEvent):
model_config = ConfigDict(arbitrary_types_allowed=True)
class MethodExecutionPausedEvent(FlowEvent):
"""Event emitted when a flow method is paused waiting for human feedback.
This event is emitted when a @human_feedback decorated method with an
async provider raises HumanFeedbackPending to pause execution.
Attributes:
flow_name: Name of the flow that is paused.
method_name: Name of the method waiting for feedback.
state: Current flow state when paused.
flow_id: Unique identifier for this flow execution.
message: The message shown when requesting feedback.
emit: Optional list of possible outcomes for routing.
"""
method_name: str
state: dict[str, Any] | BaseModel
flow_id: str
message: str
emit: list[str] | None = None
type: str = "method_execution_paused"
class FlowFinishedEvent(FlowEvent):
"""Event emitted when a flow completes execution"""
@@ -67,8 +90,71 @@ class FlowFinishedEvent(FlowEvent):
state: dict[str, Any] | BaseModel
class FlowPausedEvent(FlowEvent):
"""Event emitted when a flow is paused waiting for human feedback.
This event is emitted when a flow is paused due to a @human_feedback
decorated method with an async provider raising HumanFeedbackPending.
Attributes:
flow_name: Name of the flow that is paused.
flow_id: Unique identifier for this flow execution.
method_name: Name of the method waiting for feedback.
state: Current flow state when paused.
message: The message shown when requesting feedback.
emit: Optional list of possible outcomes for routing.
"""
flow_id: str
method_name: str
state: dict[str, Any] | BaseModel
message: str
emit: list[str] | None = None
type: str = "flow_paused"
class FlowPlotEvent(FlowEvent):
"""Event emitted when a flow plot is created"""
flow_name: str
type: str = "flow_plot"
class HumanFeedbackRequestedEvent(FlowEvent):
"""Event emitted when human feedback is requested.
This event is emitted when a @human_feedback decorated method
requires input from a human reviewer.
Attributes:
flow_name: Name of the flow requesting feedback.
method_name: Name of the method decorated with @human_feedback.
output: The method output shown to the human for review.
message: The message displayed when requesting feedback.
emit: Optional list of possible outcomes for routing.
"""
method_name: str
output: Any
message: str
emit: list[str] | None = None
type: str = "human_feedback_requested"
class HumanFeedbackReceivedEvent(FlowEvent):
"""Event emitted when human feedback is received.
This event is emitted after a human provides feedback in response
to a @human_feedback decorated method.
Attributes:
flow_name: Name of the flow that received feedback.
method_name: Name of the method that received feedback.
feedback: The raw text feedback provided by the human.
outcome: The collapsed outcome string (if emit was specified).
"""
method_name: str
feedback: str
outcome: str | None = None
type: str = "human_feedback_received"

View File

@@ -453,41 +453,48 @@ To enable tracing, do any one of these:
if flow_tree is None:
return
# Determine status-specific labels and styles
if status == "completed":
label_prefix = "✅ Flow Finished:"
style = "green"
node_text = "✅ Flow Completed"
content_text = "Flow Execution Completed"
panel_title = "Flow Completion"
elif status == "paused":
label_prefix = "⏳ Flow Paused:"
style = "cyan"
node_text = "⏳ Waiting for Human Feedback"
content_text = "Flow Paused - Waiting for Feedback"
panel_title = "Flow Paused"
else:
label_prefix = "❌ Flow Failed:"
style = "red"
node_text = "❌ Flow Failed"
content_text = "Flow Execution Failed"
panel_title = "Flow Failure"
# Update main flow label
self.update_tree_label(
flow_tree,
"✅ Flow Finished:" if status == "completed" else "❌ Flow Failed:",
label_prefix,
flow_name,
"green" if status == "completed" else "red",
style,
)
# Update initialization node status
for child in flow_tree.children:
if "Starting Flow" in str(child.label):
child.label = Text(
(
"✅ Flow Completed"
if status == "completed"
else "❌ Flow Failed"
),
style="green" if status == "completed" else "red",
)
child.label = Text(node_text, style=style)
break
content = self.create_status_content(
(
"Flow Execution Completed"
if status == "completed"
else "Flow Execution Failed"
),
content_text,
flow_name,
"green" if status == "completed" else "red",
style,
ID=flow_id,
)
self.print(flow_tree)
self.print_panel(
content, "Flow Completion", "green" if status == "completed" else "red"
)
self.print_panel(content, panel_title, style)
def update_method_status(
self,
@@ -508,6 +515,12 @@ To enable tracing, do any one of these:
if "Starting Flow" in str(child.label):
child.label = Text("Flow Method Step", style="white")
break
elif status == "paused":
prefix, style = "⏳ Paused:", "cyan"
for child in flow_tree.children:
if "Starting Flow" in str(child.label):
child.label = Text("⏳ Waiting for Feedback", style="cyan")
break
else:
prefix, style = "❌ Failed:", "red"
for child in flow_tree.children:

View File

@@ -1,3 +1,4 @@
from crewai.experimental.crew_agent_executor_flow import CrewAgentExecutorFlow
from crewai.experimental.evaluation import (
AgentEvaluationResult,
AgentEvaluator,
@@ -23,6 +24,7 @@ __all__ = [
"AgentEvaluationResult",
"AgentEvaluator",
"BaseEvaluator",
"CrewAgentExecutorFlow",
"EvaluationScore",
"EvaluationTraceCallback",
"ExperimentResult",

View File

@@ -0,0 +1,808 @@
from __future__ import annotations
from collections.abc import Callable
import threading
from typing import TYPE_CHECKING, Any, Literal, cast
from uuid import uuid4
from pydantic import BaseModel, Field, GetCoreSchemaHandler
from pydantic_core import CoreSchema, core_schema
from rich.console import Console
from rich.text import Text
from crewai.agents.agent_builder.base_agent_executor_mixin import CrewAgentExecutorMixin
from crewai.agents.parser import (
AgentAction,
AgentFinish,
OutputParserError,
)
from crewai.events.event_bus import crewai_event_bus
from crewai.events.types.logging_events import (
AgentLogsExecutionEvent,
AgentLogsStartedEvent,
)
from crewai.flow.flow import Flow, listen, or_, router, start
from crewai.hooks.llm_hooks import (
get_after_llm_call_hooks,
get_before_llm_call_hooks,
)
from crewai.utilities.agent_utils import (
enforce_rpm_limit,
format_message_for_llm,
get_llm_response,
handle_agent_action_core,
handle_context_length,
handle_max_iterations_exceeded,
handle_output_parser_exception,
handle_unknown_error,
has_reached_max_iterations,
is_context_length_exceeded,
process_llm_response,
)
from crewai.utilities.constants import TRAINING_DATA_FILE
from crewai.utilities.i18n import I18N, get_i18n
from crewai.utilities.printer import Printer
from crewai.utilities.tool_utils import execute_tool_and_check_finality
from crewai.utilities.training_handler import CrewTrainingHandler
from crewai.utilities.types import LLMMessage
if TYPE_CHECKING:
from crewai.agent import Agent
from crewai.agents.tools_handler import ToolsHandler
from crewai.crew import Crew
from crewai.llms.base_llm import BaseLLM
from crewai.task import Task
from crewai.tools.base_tool import BaseTool
from crewai.tools.structured_tool import CrewStructuredTool
from crewai.tools.tool_types import ToolResult
from crewai.utilities.prompts import StandardPromptResult, SystemPromptResult
class AgentReActState(BaseModel):
"""Structured state for agent ReAct flow execution.
Replaces scattered instance variables with validated immutable state.
Maps to: self.messages, self.iterations, formatted_answer in current executor.
"""
messages: list[LLMMessage] = Field(default_factory=list)
iterations: int = Field(default=0)
current_answer: AgentAction | AgentFinish | None = Field(default=None)
is_finished: bool = Field(default=False)
ask_for_human_input: bool = Field(default=False)
class CrewAgentExecutorFlow(Flow[AgentReActState], CrewAgentExecutorMixin):
"""Flow-based executor matching CrewAgentExecutor interface.
Inherits from:
- Flow[AgentReActState]: Provides flow orchestration capabilities
- CrewAgentExecutorMixin: Provides memory methods (short/long/external term)
Note: Multiple instances may be created during agent initialization
(cache setup, RPM controller setup, etc.) but only the final instance
should execute tasks via invoke().
"""
def __init__(
self,
llm: BaseLLM,
task: Task,
crew: Crew,
agent: Agent,
prompt: SystemPromptResult | StandardPromptResult,
max_iter: int,
tools: list[CrewStructuredTool],
tools_names: str,
stop_words: list[str],
tools_description: str,
tools_handler: ToolsHandler,
step_callback: Any = None,
original_tools: list[BaseTool] | None = None,
function_calling_llm: BaseLLM | Any | None = None,
respect_context_window: bool = False,
request_within_rpm_limit: Callable[[], bool] | None = None,
callbacks: list[Any] | None = None,
response_model: type[BaseModel] | None = None,
i18n: I18N | None = None,
) -> None:
"""Initialize the flow-based agent executor.
Args:
llm: Language model instance.
task: Task to execute.
crew: Crew instance.
agent: Agent to execute.
prompt: Prompt templates.
max_iter: Maximum iterations.
tools: Available tools.
tools_names: Tool names string.
stop_words: Stop word list.
tools_description: Tool descriptions.
tools_handler: Tool handler instance.
step_callback: Optional step callback.
original_tools: Original tool list.
function_calling_llm: Optional function calling LLM.
respect_context_window: Respect context limits.
request_within_rpm_limit: RPM limit check function.
callbacks: Optional callbacks list.
response_model: Optional Pydantic model for structured outputs.
"""
self._i18n: I18N = i18n or get_i18n()
self.llm = llm
self.task = task
self.agent = agent
self.crew = crew
self.prompt = prompt
self.tools = tools
self.tools_names = tools_names
self.stop = stop_words
self.max_iter = max_iter
self.callbacks = callbacks or []
self._printer: Printer = Printer()
self.tools_handler = tools_handler
self.original_tools = original_tools or []
self.step_callback = step_callback
self.tools_description = tools_description
self.function_calling_llm = function_calling_llm
self.respect_context_window = respect_context_window
self.request_within_rpm_limit = request_within_rpm_limit
self.response_model = response_model
self.log_error_after = 3
self._console: Console = Console()
# Error context storage for recovery
self._last_parser_error: OutputParserError | None = None
self._last_context_error: Exception | None = None
# Execution guard to prevent concurrent/duplicate executions
self._execution_lock = threading.Lock()
self._is_executing: bool = False
self._has_been_invoked: bool = False
self._flow_initialized: bool = False
self._instance_id = str(uuid4())[:8]
self.before_llm_call_hooks: list[Callable] = []
self.after_llm_call_hooks: list[Callable] = []
self.before_llm_call_hooks.extend(get_before_llm_call_hooks())
self.after_llm_call_hooks.extend(get_after_llm_call_hooks())
if self.llm:
existing_stop = getattr(self.llm, "stop", [])
self.llm.stop = list(
set(
existing_stop + self.stop
if isinstance(existing_stop, list)
else self.stop
)
)
self._state = AgentReActState()
def _ensure_flow_initialized(self) -> None:
"""Ensure Flow.__init__() has been called.
This is deferred from __init__ to prevent FlowCreatedEvent emission
during agent setup when multiple executor instances are created.
Only the instance that actually executes via invoke() will emit events.
"""
if not self._flow_initialized:
# Now call Flow's __init__ which will replace self._state
# with Flow's managed state. Suppress flow events since this is
# an agent executor, not a user-facing flow.
super().__init__(
suppress_flow_events=True,
)
self._flow_initialized = True
@property
def use_stop_words(self) -> bool:
"""Check to determine if stop words are being used.
Returns:
bool: True if stop words should be used.
"""
return self.llm.supports_stop_words() if self.llm else False
@property
def state(self) -> AgentReActState:
"""Get state - returns temporary state if Flow not yet initialized.
Flow initialization is deferred to prevent event emission during agent setup.
Returns the temporary state until invoke() is called.
"""
return self._state
@property
def messages(self) -> list[LLMMessage]:
"""Compatibility property for mixin - returns state messages."""
return self._state.messages
@property
def iterations(self) -> int:
"""Compatibility property for mixin - returns state iterations."""
return self._state.iterations
@start()
def initialize_reasoning(self) -> Literal["initialized"]:
"""Initialize the reasoning flow and emit agent start logs."""
self._show_start_logs()
return "initialized"
@listen("force_final_answer")
def force_final_answer(self) -> Literal["agent_finished"]:
"""Force agent to provide final answer when max iterations exceeded."""
formatted_answer = handle_max_iterations_exceeded(
formatted_answer=None,
printer=self._printer,
i18n=self._i18n,
messages=list(self.state.messages),
llm=self.llm,
callbacks=self.callbacks,
)
self.state.current_answer = formatted_answer
self.state.is_finished = True
return "agent_finished"
@listen("continue_reasoning")
def call_llm_and_parse(self) -> Literal["parsed", "parser_error", "context_error"]:
"""Execute LLM call with hooks and parse the response.
Returns routing decision based on parsing result.
"""
try:
enforce_rpm_limit(self.request_within_rpm_limit)
answer = get_llm_response(
llm=self.llm,
messages=list(self.state.messages),
callbacks=self.callbacks,
printer=self._printer,
from_task=self.task,
from_agent=self.agent,
response_model=self.response_model,
executor_context=self,
)
# Parse the LLM response
formatted_answer = process_llm_response(answer, self.use_stop_words)
self.state.current_answer = formatted_answer
if "Final Answer:" in answer and isinstance(formatted_answer, AgentAction):
warning_text = Text()
warning_text.append("⚠️ ", style="yellow bold")
warning_text.append(
f"LLM returned 'Final Answer:' but parsed as AgentAction (tool: {formatted_answer.tool})",
style="yellow",
)
self._console.print(warning_text)
preview_text = Text()
preview_text.append("Answer preview: ", style="yellow")
preview_text.append(f"{answer[:200]}...", style="yellow dim")
self._console.print(preview_text)
return "parsed"
except OutputParserError as e:
# Store error context for recovery
self._last_parser_error = e or OutputParserError(
error="Unknown parser error"
)
return "parser_error"
except Exception as e:
if is_context_length_exceeded(e):
self._last_context_error = e
return "context_error"
if e.__class__.__module__.startswith("litellm"):
raise e
handle_unknown_error(self._printer, e)
raise
@router(call_llm_and_parse)
def route_by_answer_type(self) -> Literal["execute_tool", "agent_finished"]:
"""Route based on whether answer is AgentAction or AgentFinish."""
if isinstance(self.state.current_answer, AgentAction):
return "execute_tool"
return "agent_finished"
@listen("execute_tool")
def execute_tool_action(self) -> Literal["tool_completed", "tool_result_is_final"]:
"""Execute the tool action and handle the result."""
try:
action = cast(AgentAction, self.state.current_answer)
# Extract fingerprint context for tool execution
fingerprint_context = {}
if (
self.agent
and hasattr(self.agent, "security_config")
and hasattr(self.agent.security_config, "fingerprint")
):
fingerprint_context = {
"agent_fingerprint": str(self.agent.security_config.fingerprint)
}
# Execute the tool
tool_result = execute_tool_and_check_finality(
agent_action=action,
fingerprint_context=fingerprint_context,
tools=self.tools,
i18n=self._i18n,
agent_key=self.agent.key if self.agent else None,
agent_role=self.agent.role if self.agent else None,
tools_handler=self.tools_handler,
task=self.task,
agent=self.agent,
function_calling_llm=self.function_calling_llm,
crew=self.crew,
)
# Handle agent action and append observation to messages
result = self._handle_agent_action(action, tool_result)
self.state.current_answer = result
# Invoke step callback if configured
self._invoke_step_callback(result)
# Append result message to conversation state
if hasattr(result, "text"):
self._append_message_to_state(result.text)
# Check if tool result became a final answer (result_as_answer flag)
if isinstance(result, AgentFinish):
self.state.is_finished = True
return "tool_result_is_final"
return "tool_completed"
except Exception as e:
error_text = Text()
error_text.append("❌ Error in tool execution: ", style="red bold")
error_text.append(str(e), style="red")
self._console.print(error_text)
raise
@listen("initialized")
def continue_iteration(self) -> Literal["check_iteration"]:
"""Bridge listener that connects iteration loop back to iteration check."""
return "check_iteration"
@router(or_(initialize_reasoning, continue_iteration))
def check_max_iterations(
self,
) -> Literal["force_final_answer", "continue_reasoning"]:
"""Check if max iterations reached before proceeding with reasoning."""
if has_reached_max_iterations(self.state.iterations, self.max_iter):
return "force_final_answer"
return "continue_reasoning"
@router(execute_tool_action)
def increment_and_continue(self) -> Literal["initialized"]:
"""Increment iteration counter and loop back for next iteration."""
self.state.iterations += 1
return "initialized"
@listen(or_("agent_finished", "tool_result_is_final"))
def finalize(self) -> Literal["completed", "skipped"]:
"""Finalize execution and emit completion logs."""
if self.state.current_answer is None:
skip_text = Text()
skip_text.append("⚠️ ", style="yellow bold")
skip_text.append(
"Finalize called but no answer in state - skipping", style="yellow"
)
self._console.print(skip_text)
return "skipped"
if not isinstance(self.state.current_answer, AgentFinish):
skip_text = Text()
skip_text.append("⚠️ ", style="yellow bold")
skip_text.append(
f"Finalize called with {type(self.state.current_answer).__name__} instead of AgentFinish - skipping",
style="yellow",
)
self._console.print(skip_text)
return "skipped"
self.state.is_finished = True
self._show_logs(self.state.current_answer)
return "completed"
@listen("parser_error")
def recover_from_parser_error(self) -> Literal["initialized"]:
"""Recover from output parser errors and retry."""
formatted_answer = handle_output_parser_exception(
e=self._last_parser_error,
messages=list(self.state.messages),
iterations=self.state.iterations,
log_error_after=self.log_error_after,
printer=self._printer,
)
if formatted_answer:
self.state.current_answer = formatted_answer
self.state.iterations += 1
return "initialized"
@listen("context_error")
def recover_from_context_length(self) -> Literal["initialized"]:
"""Recover from context length errors and retry."""
handle_context_length(
respect_context_window=self.respect_context_window,
printer=self._printer,
messages=self.state.messages,
llm=self.llm,
callbacks=self.callbacks,
i18n=self._i18n,
)
self.state.iterations += 1
return "initialized"
def invoke(self, inputs: dict[str, Any]) -> dict[str, Any]:
"""Execute agent with given inputs.
Args:
inputs: Input dictionary containing prompt variables.
Returns:
Dictionary with agent output.
"""
self._ensure_flow_initialized()
with self._execution_lock:
if self._is_executing:
raise RuntimeError(
"Executor is already running. "
"Cannot invoke the same executor instance concurrently."
)
self._is_executing = True
self._has_been_invoked = True
try:
# Reset state for fresh execution
self.state.messages.clear()
self.state.iterations = 0
self.state.current_answer = None
self.state.is_finished = False
if "system" in self.prompt:
prompt = cast("SystemPromptResult", self.prompt)
system_prompt = self._format_prompt(prompt["system"], inputs)
user_prompt = self._format_prompt(prompt["user"], inputs)
self.state.messages.append(
format_message_for_llm(system_prompt, role="system")
)
self.state.messages.append(format_message_for_llm(user_prompt))
else:
user_prompt = self._format_prompt(self.prompt["prompt"], inputs)
self.state.messages.append(format_message_for_llm(user_prompt))
self.state.ask_for_human_input = bool(
inputs.get("ask_for_human_input", False)
)
self.kickoff()
formatted_answer = self.state.current_answer
if not isinstance(formatted_answer, AgentFinish):
raise RuntimeError(
"Agent execution ended without reaching a final answer."
)
if self.state.ask_for_human_input:
formatted_answer = self._handle_human_feedback(formatted_answer)
self._create_short_term_memory(formatted_answer)
self._create_long_term_memory(formatted_answer)
self._create_external_memory(formatted_answer)
return {"output": formatted_answer.output}
except AssertionError:
fail_text = Text()
fail_text.append("", style="red bold")
fail_text.append(
"Agent failed to reach a final answer. This is likely a bug - please report it.",
style="red",
)
self._console.print(fail_text)
raise
except Exception as e:
handle_unknown_error(self._printer, e)
raise
finally:
self._is_executing = False
def _handle_agent_action(
self, formatted_answer: AgentAction, tool_result: ToolResult
) -> AgentAction | AgentFinish:
"""Process agent action and tool execution result.
Args:
formatted_answer: Agent's action to execute.
tool_result: Result from tool execution.
Returns:
Updated action or final answer.
"""
add_image_tool = self._i18n.tools("add_image")
if (
isinstance(add_image_tool, dict)
and formatted_answer.tool.casefold().strip()
== add_image_tool.get("name", "").casefold().strip()
):
self.state.messages.append(
{"role": "assistant", "content": tool_result.result}
)
return formatted_answer
return handle_agent_action_core(
formatted_answer=formatted_answer,
tool_result=tool_result,
messages=self.state.messages,
step_callback=self.step_callback,
show_logs=self._show_logs,
)
def _invoke_step_callback(
self, formatted_answer: AgentAction | AgentFinish
) -> None:
"""Invoke step callback if configured.
Args:
formatted_answer: Current agent response.
"""
if self.step_callback:
self.step_callback(formatted_answer)
def _append_message_to_state(
self, text: str, role: Literal["user", "assistant", "system"] = "assistant"
) -> None:
"""Add message to state conversation history.
Args:
text: Message content.
role: Message role (default: assistant).
"""
self.state.messages.append(format_message_for_llm(text, role=role))
def _show_start_logs(self) -> None:
"""Emit agent start event."""
if self.agent is None:
raise ValueError("Agent cannot be None")
crewai_event_bus.emit(
self.agent,
AgentLogsStartedEvent(
agent_role=self.agent.role,
task_description=(self.task.description if self.task else "Not Found"),
verbose=self.agent.verbose
or (hasattr(self, "crew") and getattr(self.crew, "verbose", False)),
),
)
def _show_logs(self, formatted_answer: AgentAction | AgentFinish) -> None:
"""Emit agent execution event.
Args:
formatted_answer: Agent's response to log.
"""
if self.agent is None:
raise ValueError("Agent cannot be None")
crewai_event_bus.emit(
self.agent,
AgentLogsExecutionEvent(
agent_role=self.agent.role,
formatted_answer=formatted_answer,
verbose=self.agent.verbose
or (hasattr(self, "crew") and getattr(self.crew, "verbose", False)),
),
)
def _handle_crew_training_output(
self, result: AgentFinish, human_feedback: str | None = None
) -> None:
"""Save training data for crew training mode.
Args:
result: Agent's final output.
human_feedback: Optional feedback from human.
"""
agent_id = str(self.agent.id)
train_iteration = (
getattr(self.crew, "_train_iteration", None) if self.crew else None
)
if train_iteration is None or not isinstance(train_iteration, int):
train_error = Text()
train_error.append("", style="red bold")
train_error.append(
"Invalid or missing train iteration. Cannot save training data.",
style="red",
)
self._console.print(train_error)
return
training_handler = CrewTrainingHandler(TRAINING_DATA_FILE)
training_data = training_handler.load() or {}
# Initialize or retrieve agent's training data
agent_training_data = training_data.get(agent_id, {})
if human_feedback is not None:
# Save initial output and human feedback
agent_training_data[train_iteration] = {
"initial_output": result.output,
"human_feedback": human_feedback,
}
else:
# Save improved output
if train_iteration in agent_training_data:
agent_training_data[train_iteration]["improved_output"] = result.output
else:
train_error = Text()
train_error.append("", style="red bold")
train_error.append(
f"No existing training data for agent {agent_id} and iteration "
f"{train_iteration}. Cannot save improved output.",
style="red",
)
self._console.print(train_error)
return
# Update the training data and save
training_data[agent_id] = agent_training_data
training_handler.save(training_data)
@staticmethod
def _format_prompt(prompt: str, inputs: dict[str, str]) -> str:
"""Format prompt template with input values.
Args:
prompt: Template string.
inputs: Values to substitute.
Returns:
Formatted prompt.
"""
prompt = prompt.replace("{input}", inputs["input"])
prompt = prompt.replace("{tool_names}", inputs["tool_names"])
return prompt.replace("{tools}", inputs["tools"])
def _handle_human_feedback(self, formatted_answer: AgentFinish) -> AgentFinish:
"""Process human feedback and refine answer.
Args:
formatted_answer: Initial agent result.
Returns:
Final answer after feedback.
"""
human_feedback = self._ask_human_input(formatted_answer.output)
if self._is_training_mode():
return self._handle_training_feedback(formatted_answer, human_feedback)
return self._handle_regular_feedback(formatted_answer, human_feedback)
def _is_training_mode(self) -> bool:
"""Check if training mode is active.
Returns:
True if in training mode.
"""
return bool(self.crew and self.crew._train)
def _handle_training_feedback(
self, initial_answer: AgentFinish, feedback: str
) -> AgentFinish:
"""Process training feedback and generate improved answer.
Args:
initial_answer: Initial agent output.
feedback: Training feedback.
Returns:
Improved answer.
"""
self._handle_crew_training_output(initial_answer, feedback)
self.state.messages.append(
format_message_for_llm(
self._i18n.slice("feedback_instructions").format(feedback=feedback)
)
)
# Re-run flow for improved answer
self.state.iterations = 0
self.state.is_finished = False
self.state.current_answer = None
self.kickoff()
# Get improved answer from state
improved_answer = self.state.current_answer
if not isinstance(improved_answer, AgentFinish):
raise RuntimeError(
"Training feedback iteration did not produce final answer"
)
self._handle_crew_training_output(improved_answer)
self.state.ask_for_human_input = False
return improved_answer
def _handle_regular_feedback(
self, current_answer: AgentFinish, initial_feedback: str
) -> AgentFinish:
"""Process regular feedback iteratively until user is satisfied.
Args:
current_answer: Current agent output.
initial_feedback: Initial user feedback.
Returns:
Final answer after iterations.
"""
feedback = initial_feedback
answer = current_answer
while self.state.ask_for_human_input:
if feedback.strip() == "":
self.state.ask_for_human_input = False
else:
answer = self._process_feedback_iteration(feedback)
feedback = self._ask_human_input(answer.output)
return answer
def _process_feedback_iteration(self, feedback: str) -> AgentFinish:
"""Process a single feedback iteration and generate updated response.
Args:
feedback: User feedback.
Returns:
Updated agent response.
"""
self.state.messages.append(
format_message_for_llm(
self._i18n.slice("feedback_instructions").format(feedback=feedback)
)
)
# Re-run flow
self.state.iterations = 0
self.state.is_finished = False
self.state.current_answer = None
self.kickoff()
# Get answer from state
answer = self.state.current_answer
if not isinstance(answer, AgentFinish):
raise RuntimeError("Feedback iteration did not produce final answer")
return answer
@classmethod
def __get_pydantic_core_schema__(
cls, _source_type: Any, _handler: GetCoreSchemaHandler
) -> CoreSchema:
"""Generate Pydantic core schema for Protocol compatibility.
Allows the executor to be used in Pydantic models without
requiring arbitrary_types_allowed=True.
"""
return core_schema.any_schema()

View File

@@ -1,8 +1,9 @@
from __future__ import annotations
from collections.abc import Sequence
import threading
from typing import Any
from typing import TYPE_CHECKING, Any
from crewai.agent.core import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.events.event_bus import crewai_event_bus
from crewai.events.types.agent_events import (
@@ -28,6 +29,10 @@ from crewai.experimental.evaluation.evaluation_listener import (
from crewai.task import Task
if TYPE_CHECKING:
from crewai.agent import Agent
class ExecutionState:
current_agent_id: str | None = None
current_task_id: str | None = None

View File

@@ -1,17 +1,22 @@
from __future__ import annotations
import abc
import enum
from enum import Enum
from typing import Any
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, Field
from crewai.agent import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.llm import BaseLLM
from crewai.task import Task
from crewai.utilities.llm_utils import create_llm
if TYPE_CHECKING:
from crewai.agent import Agent
class MetricCategory(enum.Enum):
GOAL_ALIGNMENT = "goal_alignment"
SEMANTIC_QUALITY = "semantic_quality"

View File

@@ -1,8 +1,9 @@
from __future__ import annotations
from collections import defaultdict
from hashlib import md5
from typing import Any
from typing import TYPE_CHECKING, Any
from crewai import Agent, Crew
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.experimental.evaluation import AgentEvaluator, create_default_evaluator
from crewai.experimental.evaluation.evaluation_display import (
@@ -17,6 +18,11 @@ from crewai.experimental.evaluation.experiment.result_display import (
)
if TYPE_CHECKING:
from crewai.agent import Agent
from crewai.crew import Crew
class ExperimentRunner:
def __init__(self, dataset: list[dict[str, Any]]):
self.dataset = dataset or []

View File

@@ -1,6 +1,7 @@
from typing import Any
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from crewai.agent import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.experimental.evaluation.base_evaluator import (
BaseEvaluator,
@@ -12,6 +13,10 @@ from crewai.task import Task
from crewai.utilities.types import LLMMessage
if TYPE_CHECKING:
from crewai.agent import Agent
class GoalAlignmentEvaluator(BaseEvaluator):
@property
def metric_category(self) -> MetricCategory:

View File

@@ -6,15 +6,16 @@ This module provides evaluator implementations for:
- Thinking-to-action ratio
"""
from __future__ import annotations
from collections.abc import Sequence
from enum import Enum
import logging
import re
from typing import Any
from typing import TYPE_CHECKING, Any
import numpy as np
from crewai.agent import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.experimental.evaluation.base_evaluator import (
BaseEvaluator,
@@ -27,6 +28,10 @@ from crewai.tasks.task_output import TaskOutput
from crewai.utilities.types import LLMMessage
if TYPE_CHECKING:
from crewai.agent import Agent
class ReasoningPatternType(Enum):
EFFICIENT = "efficient" # Good reasoning flow
LOOP = "loop" # Agent is stuck in a loop

View File

@@ -1,6 +1,7 @@
from typing import Any
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from crewai.agent import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.experimental.evaluation.base_evaluator import (
BaseEvaluator,
@@ -12,6 +13,10 @@ from crewai.task import Task
from crewai.utilities.types import LLMMessage
if TYPE_CHECKING:
from crewai.agent import Agent
class SemanticQualityEvaluator(BaseEvaluator):
@property
def metric_category(self) -> MetricCategory:

View File

@@ -1,7 +1,8 @@
import json
from typing import Any
from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any
from crewai.agent import Agent
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.experimental.evaluation.base_evaluator import (
BaseEvaluator,
@@ -13,6 +14,10 @@ from crewai.task import Task
from crewai.utilities.types import LLMMessage
if TYPE_CHECKING:
from crewai.agent import Agent
class ToolSelectionEvaluator(BaseEvaluator):
@property
def metric_category(self) -> MetricCategory:

View File

@@ -1,4 +1,11 @@
from crewai.flow.async_feedback import (
ConsoleProvider,
HumanFeedbackPending,
HumanFeedbackProvider,
PendingFeedbackContext,
)
from crewai.flow.flow import Flow, and_, listen, or_, router, start
from crewai.flow.human_feedback import HumanFeedbackResult, human_feedback
from crewai.flow.persistence import persist
from crewai.flow.visualization import (
FlowStructure,
@@ -8,10 +15,16 @@ from crewai.flow.visualization import (
__all__ = [
"ConsoleProvider",
"Flow",
"FlowStructure",
"HumanFeedbackPending",
"HumanFeedbackProvider",
"HumanFeedbackResult",
"PendingFeedbackContext",
"and_",
"build_flow_structure",
"human_feedback",
"listen",
"or_",
"persist",

View File

@@ -0,0 +1,41 @@
"""Async human feedback support for CrewAI Flows.
This module provides abstractions for non-blocking human-in-the-loop workflows,
allowing integration with external systems like Slack, Teams, webhooks, or APIs.
Example:
```python
from crewai.flow import Flow, start, human_feedback
from crewai.flow.async_feedback import HumanFeedbackProvider, HumanFeedbackPending
class SlackProvider(HumanFeedbackProvider):
def request_feedback(self, context, flow):
self.send_slack_notification(context)
raise HumanFeedbackPending(context=context)
class MyFlow(Flow):
@start()
@human_feedback(
message="Review this:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=SlackProvider(),
)
def review(self):
return "Content to review"
```
"""
from crewai.flow.async_feedback.types import (
HumanFeedbackPending,
HumanFeedbackProvider,
PendingFeedbackContext,
)
from crewai.flow.async_feedback.providers import ConsoleProvider
__all__ = [
"ConsoleProvider",
"HumanFeedbackPending",
"HumanFeedbackProvider",
"PendingFeedbackContext",
]

View File

@@ -0,0 +1,124 @@
"""Default provider implementations for human feedback.
This module provides the ConsoleProvider, which is the default synchronous
provider that collects feedback via console input.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from crewai.flow.async_feedback.types import PendingFeedbackContext
if TYPE_CHECKING:
from crewai.flow.flow import Flow
class ConsoleProvider:
"""Default synchronous console-based feedback provider.
This provider blocks execution and waits for console input from the user.
It displays the method output with formatting and prompts for feedback.
This is the default provider used when no custom provider is specified
in the @human_feedback decorator.
Example:
```python
from crewai.flow.async_feedback import ConsoleProvider
# Explicitly use console provider
@human_feedback(
message="Review this:",
provider=ConsoleProvider(),
)
def my_method(self):
return "Content to review"
```
"""
def __init__(self, verbose: bool = True):
"""Initialize the console provider.
Args:
verbose: Whether to display formatted output. If False, only
shows the prompt message.
"""
self.verbose = verbose
def request_feedback(
self,
context: PendingFeedbackContext,
flow: Flow,
) -> str:
"""Request feedback via console input (blocking).
Displays the method output with formatting and waits for the user
to type their feedback. Press Enter to skip (returns empty string).
Args:
context: The pending feedback context with output and message.
flow: The Flow instance (used for event emission).
Returns:
The user's feedback as a string, or empty string if skipped.
"""
from crewai.events.event_bus import crewai_event_bus
from crewai.events.event_listener import event_listener
from crewai.events.types.flow_events import (
HumanFeedbackReceivedEvent,
HumanFeedbackRequestedEvent,
)
# Emit feedback requested event
crewai_event_bus.emit(
flow,
HumanFeedbackRequestedEvent(
type="human_feedback_requested",
flow_name=flow.name or flow.__class__.__name__,
method_name=context.method_name,
output=context.method_output,
message=context.message,
emit=context.emit,
),
)
# Pause live updates during human input
formatter = event_listener.formatter
formatter.pause_live_updates()
try:
console = formatter.console
if self.verbose:
# Display output with formatting using Rich console
console.print("\n" + "" * 50, style="bold cyan")
console.print(" OUTPUT FOR REVIEW", style="bold cyan")
console.print("" * 50 + "\n", style="bold cyan")
console.print(context.method_output)
console.print("\n" + "" * 50 + "\n", style="bold cyan")
# Show message and prompt for feedback
console.print(context.message, style="yellow")
console.print(
"(Press Enter to skip, or type your feedback)\n", style="cyan"
)
feedback = input("Your feedback: ").strip()
# Emit feedback received event
crewai_event_bus.emit(
flow,
HumanFeedbackReceivedEvent(
type="human_feedback_received",
flow_name=flow.name or flow.__class__.__name__,
method_name=context.method_name,
feedback=feedback,
outcome=None, # Will be determined after collapsing
),
)
return feedback
finally:
# Resume live updates
formatter.resume_live_updates()

View File

@@ -0,0 +1,264 @@
"""Core types for async human feedback in Flows.
This module defines the protocol, exception, and context types used for
non-blocking human-in-the-loop workflows.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
if TYPE_CHECKING:
from crewai.flow.flow import Flow
@dataclass
class PendingFeedbackContext:
"""Context capturing everything needed to resume a paused flow.
When a flow is paused waiting for async human feedback, this dataclass
stores all the information needed to:
1. Identify which flow execution is waiting
2. What method triggered the feedback request
3. What was shown to the human
4. How to route the response when it arrives
Attributes:
flow_id: Unique identifier for the flow instance (from state.id)
flow_class: Fully qualified class name (e.g., "myapp.flows.ReviewFlow")
method_name: Name of the method that triggered feedback request
method_output: The output that was shown to the human for review
message: The message displayed when requesting feedback
emit: Optional list of outcome strings for routing
default_outcome: Outcome to use when no feedback is provided
metadata: Optional metadata for external system integration
llm: LLM model string for outcome collapsing
requested_at: When the feedback was requested
Example:
```python
context = PendingFeedbackContext(
flow_id="abc-123",
flow_class="myapp.ReviewFlow",
method_name="review_content",
method_output={"title": "Draft", "body": "..."},
message="Please review and approve or reject:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
```
"""
flow_id: str
flow_class: str
method_name: str
method_output: Any
message: str
emit: list[str] | None = None
default_outcome: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
llm: str | None = None
requested_at: datetime = field(default_factory=datetime.now)
def to_dict(self) -> dict[str, Any]:
"""Serialize context to a dictionary for persistence.
Returns:
Dictionary representation suitable for JSON serialization.
"""
return {
"flow_id": self.flow_id,
"flow_class": self.flow_class,
"method_name": self.method_name,
"method_output": self.method_output,
"message": self.message,
"emit": self.emit,
"default_outcome": self.default_outcome,
"metadata": self.metadata,
"llm": self.llm,
"requested_at": self.requested_at.isoformat(),
}
@classmethod
def from_dict(cls, data: dict[str, Any]) -> PendingFeedbackContext:
"""Deserialize context from a dictionary.
Args:
data: Dictionary representation of the context.
Returns:
Reconstructed PendingFeedbackContext instance.
"""
requested_at = data.get("requested_at")
if isinstance(requested_at, str):
requested_at = datetime.fromisoformat(requested_at)
elif requested_at is None:
requested_at = datetime.now()
return cls(
flow_id=data["flow_id"],
flow_class=data["flow_class"],
method_name=data["method_name"],
method_output=data.get("method_output"),
message=data.get("message", ""),
emit=data.get("emit"),
default_outcome=data.get("default_outcome"),
metadata=data.get("metadata", {}),
llm=data.get("llm"),
requested_at=requested_at,
)
class HumanFeedbackPending(Exception): # noqa: N818 - Not an error, a control flow signal
"""Signal that flow execution should pause for async human feedback.
When raised by a provider, the flow framework will:
1. Stop execution at the current method
2. Automatically persist state and context (if persistence is configured)
3. Return this object to the caller (not re-raise it)
The caller receives this as a return value from `flow.kickoff()`, enabling
graceful handling of the paused state without try/except blocks:
```python
result = flow.kickoff()
if isinstance(result, HumanFeedbackPending):
# Flow is paused, handle async feedback
print(f"Waiting for feedback: {result.context.flow_id}")
else:
# Normal completion
print(f"Flow completed: {result}")
```
Note:
The flow framework automatically saves pending feedback when this
exception is raised. Providers do NOT need to call `save_pending_feedback`
manually - just raise this exception and the framework handles persistence.
Attributes:
context: The PendingFeedbackContext with all details needed to resume
callback_info: Optional dict with information for external systems
(e.g., webhook URL, ticket ID, Slack thread ID)
Example:
```python
class SlackProvider(HumanFeedbackProvider):
def request_feedback(self, context, flow):
# Send notification to external system
ticket_id = self.create_slack_thread(context)
# Raise to pause - framework handles persistence automatically
raise HumanFeedbackPending(
context=context,
callback_info={
"slack_channel": "#reviews",
"thread_id": ticket_id,
}
)
```
"""
def __init__(
self,
context: PendingFeedbackContext,
callback_info: dict[str, Any] | None = None,
message: str | None = None,
):
"""Initialize the pending feedback exception.
Args:
context: The pending feedback context with flow details
callback_info: Optional information for external system callbacks
message: Optional custom message (defaults to descriptive message)
"""
self.context = context
self.callback_info = callback_info or {}
if message is None:
message = (
f"Human feedback pending for flow '{context.flow_id}' "
f"at method '{context.method_name}'"
)
super().__init__(message)
@runtime_checkable
class HumanFeedbackProvider(Protocol):
"""Protocol for human feedback collection strategies.
Implement this protocol to create custom feedback providers that integrate
with external systems like Slack, Teams, email, or custom APIs.
Providers can be either:
- **Synchronous (blocking)**: Return feedback string directly
- **Asynchronous (non-blocking)**: Raise HumanFeedbackPending to pause
The default ConsoleProvider is synchronous and blocks waiting for input.
For async workflows, implement a provider that raises HumanFeedbackPending.
Note:
The flow framework automatically handles state persistence when
HumanFeedbackPending is raised. Providers only need to:
1. Notify the external system (Slack, email, webhook, etc.)
2. Raise HumanFeedbackPending with the context and callback info
Example synchronous provider:
```python
class ConsoleProvider(HumanFeedbackProvider):
def request_feedback(self, context, flow):
print(context.method_output)
return input("Your feedback: ")
```
Example async provider:
```python
class SlackProvider(HumanFeedbackProvider):
def __init__(self, channel: str):
self.channel = channel
def request_feedback(self, context, flow):
# Send notification to Slack
thread_id = self.post_to_slack(
channel=self.channel,
message=context.message,
content=context.method_output,
)
# Raise to pause - framework handles persistence automatically
raise HumanFeedbackPending(
context=context,
callback_info={
"channel": self.channel,
"thread_id": thread_id,
}
)
```
"""
def request_feedback(
self,
context: PendingFeedbackContext,
flow: Flow,
) -> str:
"""Request feedback from a human.
For synchronous providers, block and return the feedback string.
For async providers, notify the external system and raise
HumanFeedbackPending to pause the flow.
Args:
context: The pending feedback context containing all details
about what feedback is needed and how to route the response.
flow: The Flow instance, providing access to state and name.
Returns:
The human's feedback as a string (synchronous providers only).
Raises:
HumanFeedbackPending: To signal that the flow should pause and
wait for external feedback. The framework will automatically
persist state when this is raised.
"""
...

File diff suppressed because it is too large Load Diff

View File

@@ -70,6 +70,15 @@ class FlowMethod(Generic[P, R]):
self._is_coroutine = asyncio.coroutines._is_coroutine # type: ignore[attr-defined]
# Preserve flow-related attributes from wrapped method (e.g., from @human_feedback)
for attr in [
"__is_router__",
"__router_paths__",
"__human_feedback_config__",
]:
if hasattr(meth, attr):
setattr(self, attr, getattr(meth, attr))
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
"""Call the wrapped method.

View File

@@ -0,0 +1,400 @@
"""Human feedback decorator for Flow methods.
This module provides the @human_feedback decorator that enables human-in-the-loop
workflows within CrewAI Flows. It allows collecting human feedback on method outputs
and optionally routing to different listeners based on the feedback.
Supports both synchronous (blocking) and asynchronous (non-blocking) feedback
collection through the provider parameter.
Example (synchronous, default):
```python
from crewai.flow import Flow, start, listen, human_feedback
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Please review this content:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def generate_content(self):
return {"title": "Article", "body": "Content..."}
@listen("approved")
def publish(self):
result = self.human_feedback
print(f"Publishing: {result.output}")
```
Example (asynchronous with custom provider):
```python
from crewai.flow import Flow, start, human_feedback
from crewai.flow.async_feedback import HumanFeedbackProvider, HumanFeedbackPending
class SlackProvider(HumanFeedbackProvider):
def request_feedback(self, context, flow):
self.send_notification(context)
raise HumanFeedbackPending(context=context)
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Review this:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=SlackProvider(),
)
def generate_content(self):
return "Content..."
```
"""
from __future__ import annotations
import asyncio
from collections.abc import Callable, Sequence
from dataclasses import dataclass, field
from datetime import datetime
from functools import wraps
from typing import TYPE_CHECKING, Any, TypeVar
from crewai.flow.flow_wrappers import FlowMethod
if TYPE_CHECKING:
from crewai.flow.async_feedback.types import HumanFeedbackProvider
from crewai.flow.flow import Flow
from crewai.llms.base_llm import BaseLLM
F = TypeVar("F", bound=Callable[..., Any])
@dataclass
class HumanFeedbackResult:
"""Result from a @human_feedback decorated method.
This dataclass captures all information about a human feedback interaction,
including the original method output, the human's feedback, and any
collapsed outcome for routing purposes.
Attributes:
output: The original return value from the decorated method that was
shown to the human for review.
feedback: The raw text feedback provided by the human. Empty string
if no feedback was provided.
outcome: The collapsed outcome string when emit is specified.
This is determined by the LLM based on the human's feedback.
None if emit was not specified.
timestamp: When the feedback was received.
method_name: The name of the decorated method that triggered feedback.
metadata: Optional metadata for enterprise integrations. Can be used
to pass additional context like channel, assignee, etc.
Example:
```python
@listen("approved")
def handle_approval(self):
result = self.human_feedback
print(f"Output: {result.output}")
print(f"Feedback: {result.feedback}")
print(f"Outcome: {result.outcome}") # "approved"
```
"""
output: Any
feedback: str
outcome: str | None = None
timestamp: datetime = field(default_factory=datetime.now)
method_name: str = ""
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass
class HumanFeedbackConfig:
"""Configuration for the @human_feedback decorator.
Stores the parameters passed to the decorator for later use during
method execution and for introspection by visualization tools.
Attributes:
message: The message shown to the human when requesting feedback.
emit: Optional sequence of outcome strings for routing.
llm: The LLM model to use for collapsing feedback to outcomes.
default_outcome: The outcome to use when no feedback is provided.
metadata: Optional metadata for enterprise integrations.
provider: Optional custom feedback provider for async workflows.
"""
message: str
emit: Sequence[str] | None = None
llm: str | BaseLLM | None = None
default_outcome: str | None = None
metadata: dict[str, Any] | None = None
provider: HumanFeedbackProvider | None = None
class HumanFeedbackMethod(FlowMethod[Any, Any]):
"""Wrapper for methods decorated with @human_feedback.
This wrapper extends FlowMethod to add human feedback specific attributes
that are used by FlowMeta for routing and by visualization tools.
Attributes:
__is_router__: True when emit is specified, enabling router behavior.
__router_paths__: List of possible outcomes when acting as a router.
__human_feedback_config__: The HumanFeedbackConfig for this method.
"""
__is_router__: bool = False
__router_paths__: list[str] | None = None
__human_feedback_config__: HumanFeedbackConfig | None = None
def human_feedback(
message: str,
emit: Sequence[str] | None = None,
llm: str | BaseLLM | None = None,
default_outcome: str | None = None,
metadata: dict[str, Any] | None = None,
provider: HumanFeedbackProvider | None = None,
) -> Callable[[F], F]:
"""Decorator for Flow methods that require human feedback.
This decorator wraps a Flow method to:
1. Execute the method and capture its output
2. Display the output to the human with a feedback request
3. Collect the human's free-form feedback
4. Optionally collapse the feedback to a predefined outcome using an LLM
5. Store the result for access by downstream methods
When `emit` is specified, the decorator acts as a router, and the
collapsed outcome triggers the appropriate @listen decorated method.
Supports both synchronous (blocking) and asynchronous (non-blocking)
feedback collection through the `provider` parameter. If no provider
is specified, defaults to synchronous console input.
Args:
message: The message shown to the human when requesting feedback.
This should clearly explain what kind of feedback is expected.
emit: Optional sequence of outcome strings. When provided, the
human's feedback will be collapsed to one of these outcomes
using the specified LLM. The outcome then triggers @listen
methods that match.
llm: The LLM model to use for collapsing feedback to outcomes.
Required when emit is specified. Can be a model string
like "gpt-4o-mini" or a BaseLLM instance.
default_outcome: The outcome to use when the human provides no
feedback (empty input). Must be one of the emit values
if emit is specified.
metadata: Optional metadata for enterprise integrations. This is
passed through to the HumanFeedbackResult and can be used
by enterprise forks for features like Slack/Teams integration.
provider: Optional HumanFeedbackProvider for custom feedback
collection. Use this for async workflows that integrate with
external systems like Slack, Teams, or webhooks. When the
provider raises HumanFeedbackPending, the flow pauses and
can be resumed later with Flow.resume().
Returns:
A decorator function that wraps the method with human feedback
collection logic.
Raises:
ValueError: If emit is specified but llm is not provided.
ValueError: If default_outcome is specified but emit is not.
ValueError: If default_outcome is not in the emit list.
HumanFeedbackPending: When an async provider pauses execution.
Example:
Basic feedback without routing:
```python
@start()
@human_feedback(message="Please review this output:")
def generate_content(self):
return "Generated content..."
```
With routing based on feedback:
```python
@start()
@human_feedback(
message="Review and approve or reject:",
emit=["approved", "rejected", "needs_revision"],
llm="gpt-4o-mini",
default_outcome="needs_revision",
)
def review_document(self):
return document_content
@listen("approved")
def publish(self):
print(f"Publishing: {self.last_human_feedback.output}")
```
Async feedback with custom provider:
```python
@start()
@human_feedback(
message="Review this content:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
provider=SlackProvider(channel="#reviews"),
)
def generate_content(self):
return "Content to review..."
```
"""
# Validation at decoration time
if emit is not None:
if not llm:
raise ValueError(
"llm is required when emit is specified. "
"Provide an LLM model string (e.g., 'gpt-4o-mini') or a BaseLLM instance."
)
if default_outcome is not None and default_outcome not in emit:
raise ValueError(
f"default_outcome '{default_outcome}' must be one of the "
f"emit options: {list(emit)}"
)
elif default_outcome is not None:
raise ValueError("default_outcome requires emit to be specified.")
def decorator(func: F) -> F:
"""Inner decorator that wraps the function."""
def _request_feedback(flow_instance: Flow, method_output: Any) -> str:
"""Request feedback using provider or default console."""
from crewai.flow.async_feedback.types import PendingFeedbackContext
# Build context for provider
# Use flow_id property which handles both dict and BaseModel states
context = PendingFeedbackContext(
flow_id=flow_instance.flow_id or "unknown",
flow_class=f"{flow_instance.__class__.__module__}.{flow_instance.__class__.__name__}",
method_name=func.__name__,
method_output=method_output,
message=message,
emit=list(emit) if emit else None,
default_outcome=default_outcome,
metadata=metadata or {},
llm=llm if isinstance(llm, str) else None,
)
if provider is not None:
# Use custom provider (may raise HumanFeedbackPending)
return provider.request_feedback(context, flow_instance)
else:
# Use default console input
return flow_instance._request_human_feedback(
message=message,
output=method_output,
metadata=metadata,
emit=emit,
)
def _process_feedback(
flow_instance: Flow,
method_output: Any,
raw_feedback: str,
) -> HumanFeedbackResult | str:
"""Process feedback and return result or outcome."""
# Determine outcome
collapsed_outcome: str | None = None
if not raw_feedback.strip():
# Empty feedback
if default_outcome:
collapsed_outcome = default_outcome
elif emit:
# No default and no feedback - use first outcome
collapsed_outcome = emit[0]
elif emit:
# Collapse feedback to outcome using LLM
collapsed_outcome = flow_instance._collapse_to_outcome(
feedback=raw_feedback,
outcomes=emit,
llm=llm,
)
# Create result
result = HumanFeedbackResult(
output=method_output,
feedback=raw_feedback,
outcome=collapsed_outcome,
timestamp=datetime.now(),
method_name=func.__name__,
metadata=metadata or {},
)
# Store in flow instance
flow_instance.human_feedback_history.append(result)
flow_instance.last_human_feedback = result
# Return based on mode
if emit:
# Return outcome for routing
return collapsed_outcome # type: ignore[return-value]
return result
if asyncio.iscoroutinefunction(func):
# Async wrapper
@wraps(func)
async def async_wrapper(self: Flow, *args: Any, **kwargs: Any) -> Any:
# Execute the original method
method_output = await func(self, *args, **kwargs)
# Request human feedback (may raise HumanFeedbackPending)
raw_feedback = _request_feedback(self, method_output)
# Process and return
return _process_feedback(self, method_output, raw_feedback)
wrapper: Any = async_wrapper
else:
# Sync wrapper
@wraps(func)
def sync_wrapper(self: Flow, *args: Any, **kwargs: Any) -> Any:
# Execute the original method
method_output = func(self, *args, **kwargs)
# Request human feedback (may raise HumanFeedbackPending)
raw_feedback = _request_feedback(self, method_output)
# Process and return
return _process_feedback(self, method_output, raw_feedback)
wrapper = sync_wrapper
# Preserve existing Flow decorator attributes
for attr in [
"__is_start_method__",
"__trigger_methods__",
"__condition_type__",
"__trigger_condition__",
"__is_flow_method__",
]:
if hasattr(func, attr):
setattr(wrapper, attr, getattr(func, attr))
# Add human feedback specific attributes (create config inline to avoid race conditions)
wrapper.__human_feedback_config__ = HumanFeedbackConfig(
message=message,
emit=emit,
llm=llm,
default_outcome=default_outcome,
metadata=metadata,
provider=provider,
)
wrapper.__is_flow_method__ = True
# Make it a router if emit specified
if emit:
wrapper.__is_router__ = True
wrapper.__router_paths__ = list(emit)
return wrapper # type: ignore[return-value]
return decorator

View File

@@ -1,16 +1,26 @@
"""Base class for flow state persistence."""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Any
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel
if TYPE_CHECKING:
from crewai.flow.async_feedback.types import PendingFeedbackContext
class FlowPersistence(ABC):
"""Abstract base class for flow state persistence.
This class defines the interface that all persistence implementations must follow.
It supports both structured (Pydantic BaseModel) and unstructured (dict) states.
For async human feedback support, implementations can optionally override:
- save_pending_feedback(): Saves state with pending feedback context
- load_pending_feedback(): Loads state and pending feedback context
- clear_pending_feedback(): Clears pending feedback after resume
"""
@abstractmethod
@@ -45,3 +55,52 @@ class FlowPersistence(ABC):
Returns:
The most recent state as a dictionary, or None if no state exists
"""
def save_pending_feedback(
self,
flow_uuid: str,
context: PendingFeedbackContext,
state_data: dict[str, Any] | BaseModel,
) -> None:
"""Save state with a pending feedback marker.
This method is called when a flow is paused waiting for async human
feedback. The default implementation just saves the state without
the pending feedback context. Override to store the context.
Args:
flow_uuid: Unique identifier for the flow instance
context: The pending feedback context with all resume information
state_data: Current state data
"""
# Default: just save the state without pending context
self.save_state(flow_uuid, context.method_name, state_data)
def load_pending_feedback(
self,
flow_uuid: str,
) -> tuple[dict[str, Any], PendingFeedbackContext] | None:
"""Load state and pending feedback context.
This method is called when resuming a paused flow. Override to
load both the state and the pending feedback context.
Args:
flow_uuid: Unique identifier for the flow instance
Returns:
Tuple of (state_data, pending_context) if pending feedback exists,
None otherwise.
"""
return None
def clear_pending_feedback(self, flow_uuid: str) -> None: # noqa: B027
"""Clear the pending feedback marker after successful resume.
This is called after feedback is received and the flow resumes.
Optional override to remove the pending feedback marker.
Args:
flow_uuid: Unique identifier for the flow instance
"""
pass

View File

@@ -2,17 +2,22 @@
SQLite-based implementation of flow state persistence.
"""
from __future__ import annotations
from datetime import datetime, timezone
import json
from pathlib import Path
import sqlite3
from typing import Any
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel
from crewai.flow.persistence.base import FlowPersistence
from crewai.utilities.paths import db_storage_path
if TYPE_CHECKING:
from crewai.flow.async_feedback.types import PendingFeedbackContext
class SQLiteFlowPersistence(FlowPersistence):
"""SQLite-based implementation of flow state persistence.
@@ -20,6 +25,28 @@ class SQLiteFlowPersistence(FlowPersistence):
This class provides a simple, file-based persistence implementation using SQLite.
It's suitable for development and testing, or for production use cases with
moderate performance requirements.
This implementation supports async human feedback by storing pending feedback
context in a separate table. When a flow is paused waiting for feedback,
use save_pending_feedback() to persist the context. Later, use
load_pending_feedback() to retrieve it when resuming.
Example:
```python
persistence = SQLiteFlowPersistence("flows.db")
# Start a flow with async feedback
try:
flow = MyFlow(persistence=persistence)
result = flow.kickoff()
except HumanFeedbackPending as e:
# Flow is paused, state is already persisted
print(f"Waiting for feedback: {e.context.flow_id}")
# Later, resume with feedback
flow = MyFlow.from_pending("abc-123", persistence)
result = flow.resume("looks good!")
```
"""
def __init__(self, db_path: str | None = None) -> None:
@@ -45,6 +72,7 @@ class SQLiteFlowPersistence(FlowPersistence):
def init_db(self) -> None:
"""Create the necessary tables if they don't exist."""
with sqlite3.connect(self.db_path) as conn:
# Main state table
conn.execute(
"""
CREATE TABLE IF NOT EXISTS flow_states (
@@ -64,6 +92,26 @@ class SQLiteFlowPersistence(FlowPersistence):
"""
)
# Pending feedback table for async HITL
conn.execute(
"""
CREATE TABLE IF NOT EXISTS pending_feedback (
id INTEGER PRIMARY KEY AUTOINCREMENT,
flow_uuid TEXT NOT NULL UNIQUE,
context_json TEXT NOT NULL,
state_json TEXT NOT NULL,
created_at DATETIME NOT NULL
)
"""
)
# Add index for faster UUID lookups on pending feedback
conn.execute(
"""
CREATE INDEX IF NOT EXISTS idx_pending_feedback_uuid
ON pending_feedback(flow_uuid)
"""
)
def save_state(
self,
flow_uuid: str,
@@ -130,3 +178,104 @@ class SQLiteFlowPersistence(FlowPersistence):
if row:
return json.loads(row[0])
return None
def save_pending_feedback(
self,
flow_uuid: str,
context: PendingFeedbackContext,
state_data: dict[str, Any] | BaseModel,
) -> None:
"""Save state with a pending feedback marker.
This method stores both the flow state and the pending feedback context,
allowing the flow to be resumed later when feedback is received.
Args:
flow_uuid: Unique identifier for the flow instance
context: The pending feedback context with all resume information
state_data: Current state data
"""
# Import here to avoid circular imports
from crewai.flow.async_feedback.types import PendingFeedbackContext
# Convert state_data to dict
if isinstance(state_data, BaseModel):
state_dict = state_data.model_dump()
elif isinstance(state_data, dict):
state_dict = state_data
else:
raise ValueError(
f"state_data must be either a Pydantic BaseModel or dict, got {type(state_data)}"
)
# Also save to regular state table for consistency
self.save_state(flow_uuid, context.method_name, state_data)
# Save pending feedback context
with sqlite3.connect(self.db_path) as conn:
# Use INSERT OR REPLACE to handle re-triggering feedback on same flow
conn.execute(
"""
INSERT OR REPLACE INTO pending_feedback (
flow_uuid,
context_json,
state_json,
created_at
) VALUES (?, ?, ?, ?)
""",
(
flow_uuid,
json.dumps(context.to_dict()),
json.dumps(state_dict),
datetime.now(timezone.utc).isoformat(),
),
)
def load_pending_feedback(
self,
flow_uuid: str,
) -> tuple[dict[str, Any], PendingFeedbackContext] | None:
"""Load state and pending feedback context.
Args:
flow_uuid: Unique identifier for the flow instance
Returns:
Tuple of (state_data, pending_context) if pending feedback exists,
None otherwise.
"""
# Import here to avoid circular imports
from crewai.flow.async_feedback.types import PendingFeedbackContext
with sqlite3.connect(self.db_path) as conn:
cursor = conn.execute(
"""
SELECT state_json, context_json
FROM pending_feedback
WHERE flow_uuid = ?
""",
(flow_uuid,),
)
row = cursor.fetchone()
if row:
state_dict = json.loads(row[0])
context_dict = json.loads(row[1])
context = PendingFeedbackContext.from_dict(context_dict)
return (state_dict, context)
return None
def clear_pending_feedback(self, flow_uuid: str) -> None:
"""Clear the pending feedback marker after successful resume.
Args:
flow_uuid: Unique identifier for the flow instance
"""
with sqlite3.connect(self.db_path) as conn:
conn.execute(
"""
DELETE FROM pending_feedback
WHERE flow_uuid = ?
""",
(flow_uuid,),
)

View File

@@ -29,7 +29,8 @@
"lite_agent_system_prompt_without_tools": "You are {role}. {backstory}\nYour personal goal is: {goal}\n\nTo give my best complete final answer to the task respond using the exact following format:\n\nThought: I now can give a great answer\nFinal Answer: Your final answer must be the great and the most complete as possible, it must be outcome described.\n\nI MUST use these formats, my job depends on it!",
"lite_agent_response_format": "Ensure your final answer strictly adheres to the following OpenAPI schema: {response_format}\n\nDo not include the OpenAPI schema in the final output. Ensure the final output does not include any code block markers like ```json or ```python.",
"knowledge_search_query": "The original query is: {task_prompt}.",
"knowledge_search_query_system_prompt": "Your goal is to rewrite the user query so that it is optimized for retrieval from a vector database. Consider how the query will be used to find relevant documents, and aim to make it more specific and context-aware. \n\n Do not include any other text than the rewritten query, especially any preamble or postamble and only add expected output format if its relevant to the rewritten query. \n\n Focus on the key words of the intended task and to retrieve the most relevant information. \n\n There will be some extra context provided that might need to be removed such as expected_output formats structured_outputs and other instructions."
"knowledge_search_query_system_prompt": "Your goal is to rewrite the user query so that it is optimized for retrieval from a vector database. Consider how the query will be used to find relevant documents, and aim to make it more specific and context-aware. \n\n Do not include any other text than the rewritten query, especially any preamble or postamble and only add expected output format if its relevant to the rewritten query. \n\n Focus on the key words of the intended task and to retrieve the most relevant information. \n\n There will be some extra context provided that might need to be removed such as expected_output formats structured_outputs and other instructions.",
"human_feedback_collapse": "Based on the following human feedback, determine which outcome best matches their intent.\n\nFeedback: {feedback}\n\nPossible outcomes: {outcomes}\n\nRespond with ONLY one of the exact outcome values listed above, nothing else."
},
"errors": {
"force_final_answer_error": "You can't keep going, here is the best final answer you generated:\n\n {formatted_answer}",

View File

@@ -1178,6 +1178,7 @@ def test_system_and_prompt_template():
{{ .Response }}<|eot_id|>""",
)
agent.create_agent_executor()
expected_prompt = """<|start_header_id|>system<|end_header_id|>
@@ -1442,6 +1443,8 @@ def test_agent_max_retry_limit():
human_input=True,
)
agent.create_agent_executor(task=task)
error_message = "Error happening while sending prompt to model."
with patch.object(
CrewAgentExecutor, "invoke", wraps=agent.agent_executor.invoke
@@ -1503,9 +1506,8 @@ def test_agent_with_custom_stop_words():
)
assert isinstance(agent.llm, BaseLLM)
assert set(agent.llm.stop) == set([*stop_words, "\nObservation:"])
assert set(agent.llm.stop) == set(stop_words)
assert all(word in agent.llm.stop for word in stop_words)
assert "\nObservation:" in agent.llm.stop
def test_agent_with_callbacks():
@@ -1629,6 +1631,8 @@ def test_handle_context_length_exceeds_limit_cli_no():
)
task = Task(description="test task", agent=agent, expected_output="test output")
agent.create_agent_executor(task=task)
with patch.object(
CrewAgentExecutor, "invoke", wraps=agent.agent_executor.invoke
) as private_mock:
@@ -1679,8 +1683,8 @@ def test_agent_with_all_llm_attributes():
assert agent.llm.temperature == 0.7
assert agent.llm.top_p == 0.9
# assert agent.llm.n == 1
assert set(agent.llm.stop) == set(["STOP", "END", "\nObservation:"])
assert all(word in agent.llm.stop for word in ["STOP", "END", "\nObservation:"])
assert set(agent.llm.stop) == set(["STOP", "END"])
assert all(word in agent.llm.stop for word in ["STOP", "END"])
assert agent.llm.max_tokens == 100
assert agent.llm.presence_penalty == 0.1
assert agent.llm.frequency_penalty == 0.1

View File

@@ -1,313 +0,0 @@
"""Test Agent timeout handling and cooperative cancellation."""
import threading
import time
from unittest.mock import MagicMock, patch
import pytest
from crewai import Agent, Task
from crewai.agents.crew_agent_executor import CrewAgentExecutor
class TestExecutorDeadline:
"""Tests for CrewAgentExecutor deadline functionality."""
def test_set_execution_deadline(self):
"""Test that set_execution_deadline sets the deadline correctly."""
executor = MagicMock(spec=CrewAgentExecutor)
executor._execution_deadline = None
CrewAgentExecutor.set_execution_deadline(executor, 5)
assert executor._execution_deadline is not None
assert executor._execution_deadline > time.monotonic()
def test_clear_execution_deadline(self):
"""Test that clear_execution_deadline clears the deadline."""
executor = MagicMock(spec=CrewAgentExecutor)
executor._execution_deadline = time.monotonic() + 100
CrewAgentExecutor.clear_execution_deadline(executor)
assert executor._execution_deadline is None
def test_check_execution_deadline_not_exceeded(self):
"""Test that _check_execution_deadline does not raise when deadline not exceeded."""
executor = MagicMock(spec=CrewAgentExecutor)
executor._execution_deadline = time.monotonic() + 100
executor.task = MagicMock()
executor.task.description = "Test task"
CrewAgentExecutor._check_execution_deadline(executor)
def test_check_execution_deadline_exceeded(self):
"""Test that _check_execution_deadline raises TimeoutError when deadline exceeded."""
executor = MagicMock(spec=CrewAgentExecutor)
executor._execution_deadline = time.monotonic() - 1
executor.task = MagicMock()
executor.task.description = "Test task"
with pytest.raises(TimeoutError) as exc_info:
CrewAgentExecutor._check_execution_deadline(executor)
assert "Test task" in str(exc_info.value)
assert "timed out" in str(exc_info.value)
def test_check_execution_deadline_no_deadline_set(self):
"""Test that _check_execution_deadline does nothing when no deadline is set."""
executor = MagicMock(spec=CrewAgentExecutor)
executor._execution_deadline = None
executor.task = MagicMock()
executor.task.description = "Test task"
CrewAgentExecutor._check_execution_deadline(executor)
class TestAgentTimeoutBehavior:
"""Tests for Agent timeout behavior."""
def test_execute_with_timeout_sets_deadline(self):
"""Test that _execute_with_timeout sets the deadline on the executor."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=5,
)
mock_executor = MagicMock()
mock_executor.invoke.return_value = {"output": "test output"}
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
with patch.object(agent, "_execute_without_timeout", return_value="test output"):
agent._execute_with_timeout("test prompt", task, 5)
mock_executor.set_execution_deadline.assert_called_once_with(5)
mock_executor.clear_execution_deadline.assert_called_once()
def test_execute_with_timeout_clears_deadline_on_success(self):
"""Test that _execute_with_timeout clears the deadline after successful execution."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=5,
)
mock_executor = MagicMock()
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
with patch.object(agent, "_execute_without_timeout", return_value="test output"):
result = agent._execute_with_timeout("test prompt", task, 5)
assert result == "test output"
mock_executor.clear_execution_deadline.assert_called_once()
def test_execute_with_timeout_clears_deadline_on_timeout(self):
"""Test that _execute_with_timeout clears the deadline even when timeout occurs."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=1,
)
mock_executor = MagicMock()
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
def slow_execution(*args, **kwargs):
time.sleep(5)
return "test output"
with patch.object(agent, "_execute_without_timeout", side_effect=slow_execution):
with pytest.raises(TimeoutError):
agent._execute_with_timeout("test prompt", task, 1)
mock_executor.clear_execution_deadline.assert_called_once()
def test_execute_with_timeout_raises_timeout_error(self):
"""Test that _execute_with_timeout raises TimeoutError when execution exceeds timeout."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=1,
)
mock_executor = MagicMock()
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
def slow_execution(*args, **kwargs):
time.sleep(5)
return "test output"
with patch.object(agent, "_execute_without_timeout", side_effect=slow_execution):
with pytest.raises(TimeoutError) as exc_info:
agent._execute_with_timeout("test prompt", task, 1)
assert "Test task" in str(exc_info.value)
assert "timed out" in str(exc_info.value)
class TestCooperativeCancellation:
"""Tests for cooperative cancellation behavior."""
def test_timeout_returns_control_promptly(self):
"""Test that timeout returns control to caller promptly (within reasonable bounds)."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=1,
)
mock_executor = MagicMock()
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
execution_started = threading.Event()
def slow_execution(*args, **kwargs):
execution_started.set()
time.sleep(10)
return "test output"
with patch.object(agent, "_execute_without_timeout", side_effect=slow_execution):
start_time = time.monotonic()
with pytest.raises(TimeoutError):
agent._execute_with_timeout("test prompt", task, 1)
elapsed_time = time.monotonic() - start_time
assert elapsed_time < 3, f"Timeout should return control within 3 seconds, took {elapsed_time:.2f}s"
def test_executor_deadline_checked_in_invoke_loop(self):
"""Test that the executor checks the deadline in the invoke loop."""
mock_llm = MagicMock()
mock_llm.supports_stop_words.return_value = False
mock_llm.call.return_value = "Final Answer: test"
mock_task = MagicMock()
mock_task.description = "Test task"
mock_crew = MagicMock()
mock_crew.verbose = False
mock_agent = MagicMock()
mock_agent.verbose = False
mock_agent.role = "Test Agent"
executor = CrewAgentExecutor(
llm=mock_llm,
task=mock_task,
crew=mock_crew,
agent=mock_agent,
prompt={"prompt": "test"},
max_iter=10,
tools=[],
tools_names="",
stop_words=[],
tools_description="",
tools_handler=MagicMock(),
)
executor.set_execution_deadline(0.001)
time.sleep(0.01)
with pytest.raises(TimeoutError) as exc_info:
executor.invoke({"input": "test", "tool_names": "", "tools": ""})
assert "timed out" in str(exc_info.value)
class TestAsyncTimeoutBehavior:
"""Tests for async timeout behavior."""
@pytest.mark.asyncio
async def test_aexecute_with_timeout_sets_deadline(self):
"""Test that _aexecute_with_timeout sets the deadline on the executor."""
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=5,
)
mock_executor = MagicMock()
mock_executor.ainvoke = MagicMock(return_value={"output": "test output"})
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
async def mock_aexecute(*args, **kwargs):
return "test output"
with patch.object(agent, "_aexecute_without_timeout", side_effect=mock_aexecute):
await agent._aexecute_with_timeout("test prompt", task, 5)
mock_executor.set_execution_deadline.assert_called_once_with(5)
mock_executor.clear_execution_deadline.assert_called_once()
@pytest.mark.asyncio
async def test_aexecute_with_timeout_clears_deadline_on_timeout(self):
"""Test that _aexecute_with_timeout clears the deadline even when timeout occurs."""
import asyncio
agent = Agent(
role="test role",
goal="test goal",
backstory="test backstory",
max_execution_time=1,
)
mock_executor = MagicMock()
agent.agent_executor = mock_executor
task = Task(
description="Test task",
expected_output="Test output",
agent=agent,
)
async def slow_execution(*args, **kwargs):
await asyncio.sleep(5)
return "test output"
with patch.object(agent, "_aexecute_without_timeout", side_effect=slow_execution):
with pytest.raises(TimeoutError):
await agent._aexecute_with_timeout("test prompt", task, 1)
mock_executor.clear_execution_deadline.assert_called_once()

View File

@@ -0,0 +1,479 @@
"""Unit tests for CrewAgentExecutorFlow.
Tests the Flow-based agent executor implementation including state management,
flow methods, routing logic, and error handling.
"""
from unittest.mock import Mock, patch
import pytest
from crewai.experimental.crew_agent_executor_flow import (
AgentReActState,
CrewAgentExecutorFlow,
)
from crewai.agents.parser import AgentAction, AgentFinish
class TestAgentReActState:
"""Test AgentReActState Pydantic model."""
def test_state_initialization(self):
"""Test AgentReActState initialization with defaults."""
state = AgentReActState()
assert state.iterations == 0
assert state.messages == []
assert state.current_answer is None
assert state.is_finished is False
assert state.ask_for_human_input is False
def test_state_with_values(self):
"""Test AgentReActState initialization with values."""
messages = [{"role": "user", "content": "test"}]
state = AgentReActState(
messages=messages,
iterations=5,
current_answer=AgentFinish(thought="thinking", output="done", text="final"),
is_finished=True,
ask_for_human_input=True,
)
assert state.messages == messages
assert state.iterations == 5
assert isinstance(state.current_answer, AgentFinish)
assert state.is_finished is True
assert state.ask_for_human_input is True
class TestCrewAgentExecutorFlow:
"""Test CrewAgentExecutorFlow class."""
@pytest.fixture
def mock_dependencies(self):
"""Create mock dependencies for executor."""
llm = Mock()
llm.supports_stop_words.return_value = True
task = Mock()
task.description = "Test task"
task.human_input = False
task.response_model = None
crew = Mock()
crew.verbose = False
crew._train = False
agent = Mock()
agent.id = "test-agent-id"
agent.role = "Test Agent"
agent.verbose = False
agent.key = "test-key"
prompt = {"prompt": "Test prompt with {input}, {tool_names}, {tools}"}
tools = []
tools_handler = Mock()
return {
"llm": llm,
"task": task,
"crew": crew,
"agent": agent,
"prompt": prompt,
"max_iter": 10,
"tools": tools,
"tools_names": "",
"stop_words": ["Observation"],
"tools_description": "",
"tools_handler": tools_handler,
}
def test_executor_initialization(self, mock_dependencies):
"""Test CrewAgentExecutorFlow initialization."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
assert executor.llm == mock_dependencies["llm"]
assert executor.task == mock_dependencies["task"]
assert executor.agent == mock_dependencies["agent"]
assert executor.crew == mock_dependencies["crew"]
assert executor.max_iter == 10
assert executor.use_stop_words is True
def test_initialize_reasoning(self, mock_dependencies):
"""Test flow entry point."""
with patch.object(
CrewAgentExecutorFlow, "_show_start_logs"
) as mock_show_start:
executor = CrewAgentExecutorFlow(**mock_dependencies)
result = executor.initialize_reasoning()
assert result == "initialized"
mock_show_start.assert_called_once()
def test_check_max_iterations_not_reached(self, mock_dependencies):
"""Test routing when iterations < max."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.iterations = 5
result = executor.check_max_iterations()
assert result == "continue_reasoning"
def test_check_max_iterations_reached(self, mock_dependencies):
"""Test routing when iterations >= max."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.iterations = 10
result = executor.check_max_iterations()
assert result == "force_final_answer"
def test_route_by_answer_type_action(self, mock_dependencies):
"""Test routing for AgentAction."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.current_answer = AgentAction(
thought="thinking", tool="search", tool_input="query", text="action text"
)
result = executor.route_by_answer_type()
assert result == "execute_tool"
def test_route_by_answer_type_finish(self, mock_dependencies):
"""Test routing for AgentFinish."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.current_answer = AgentFinish(
thought="final thoughts", output="Final answer", text="complete"
)
result = executor.route_by_answer_type()
assert result == "agent_finished"
def test_continue_iteration(self, mock_dependencies):
"""Test iteration continuation."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
result = executor.continue_iteration()
assert result == "check_iteration"
def test_finalize_success(self, mock_dependencies):
"""Test finalize with valid AgentFinish."""
with patch.object(CrewAgentExecutorFlow, "_show_logs") as mock_show_logs:
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.current_answer = AgentFinish(
thought="final thinking", output="Done", text="complete"
)
result = executor.finalize()
assert result == "completed"
assert executor.state.is_finished is True
mock_show_logs.assert_called_once()
def test_finalize_failure(self, mock_dependencies):
"""Test finalize skips when given AgentAction instead of AgentFinish."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.current_answer = AgentAction(
thought="thinking", tool="search", tool_input="query", text="action text"
)
result = executor.finalize()
# Should return "skipped" and not set is_finished
assert result == "skipped"
assert executor.state.is_finished is False
def test_format_prompt(self, mock_dependencies):
"""Test prompt formatting."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
inputs = {"input": "test input", "tool_names": "tool1, tool2", "tools": "desc"}
result = executor._format_prompt("Prompt {input} {tool_names} {tools}", inputs)
assert "test input" in result
assert "tool1, tool2" in result
assert "desc" in result
def test_is_training_mode_false(self, mock_dependencies):
"""Test training mode detection when not in training."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
assert executor._is_training_mode() is False
def test_is_training_mode_true(self, mock_dependencies):
"""Test training mode detection when in training."""
mock_dependencies["crew"]._train = True
executor = CrewAgentExecutorFlow(**mock_dependencies)
assert executor._is_training_mode() is True
def test_append_message_to_state(self, mock_dependencies):
"""Test message appending to state."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
initial_count = len(executor.state.messages)
executor._append_message_to_state("test message")
assert len(executor.state.messages) == initial_count + 1
assert executor.state.messages[-1]["content"] == "test message"
def test_invoke_step_callback(self, mock_dependencies):
"""Test step callback invocation."""
callback = Mock()
mock_dependencies["step_callback"] = callback
executor = CrewAgentExecutorFlow(**mock_dependencies)
answer = AgentFinish(thought="thinking", output="test", text="final")
executor._invoke_step_callback(answer)
callback.assert_called_once_with(answer)
def test_invoke_step_callback_none(self, mock_dependencies):
"""Test step callback when none provided."""
mock_dependencies["step_callback"] = None
executor = CrewAgentExecutorFlow(**mock_dependencies)
# Should not raise error
executor._invoke_step_callback(
AgentFinish(thought="thinking", output="test", text="final")
)
@patch("crewai.experimental.crew_agent_executor_flow.handle_output_parser_exception")
def test_recover_from_parser_error(
self, mock_handle_exception, mock_dependencies
):
"""Test recovery from OutputParserError."""
from crewai.agents.parser import OutputParserError
mock_handle_exception.return_value = None
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor._last_parser_error = OutputParserError("test error")
initial_iterations = executor.state.iterations
result = executor.recover_from_parser_error()
assert result == "initialized"
assert executor.state.iterations == initial_iterations + 1
mock_handle_exception.assert_called_once()
@patch("crewai.experimental.crew_agent_executor_flow.handle_context_length")
def test_recover_from_context_length(
self, mock_handle_context, mock_dependencies
):
"""Test recovery from context length error."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor._last_context_error = Exception("context too long")
initial_iterations = executor.state.iterations
result = executor.recover_from_context_length()
assert result == "initialized"
assert executor.state.iterations == initial_iterations + 1
mock_handle_context.assert_called_once()
def test_use_stop_words_property(self, mock_dependencies):
"""Test use_stop_words property."""
mock_dependencies["llm"].supports_stop_words.return_value = True
executor = CrewAgentExecutorFlow(**mock_dependencies)
assert executor.use_stop_words is True
mock_dependencies["llm"].supports_stop_words.return_value = False
executor = CrewAgentExecutorFlow(**mock_dependencies)
assert executor.use_stop_words is False
def test_compatibility_properties(self, mock_dependencies):
"""Test compatibility properties for mixin."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.messages = [{"role": "user", "content": "test"}]
executor.state.iterations = 5
# Test that compatibility properties return state values
assert executor.messages == executor.state.messages
assert executor.iterations == executor.state.iterations
class TestFlowErrorHandling:
"""Test error handling in flow methods."""
@pytest.fixture
def mock_dependencies(self):
"""Create mock dependencies."""
llm = Mock()
llm.supports_stop_words.return_value = True
task = Mock()
task.description = "Test task"
crew = Mock()
agent = Mock()
agent.role = "Test Agent"
agent.verbose = False
prompt = {"prompt": "Test {input}"}
return {
"llm": llm,
"task": task,
"crew": crew,
"agent": agent,
"prompt": prompt,
"max_iter": 10,
"tools": [],
"tools_names": "",
"stop_words": [],
"tools_description": "",
"tools_handler": Mock(),
}
@patch("crewai.experimental.crew_agent_executor_flow.get_llm_response")
@patch("crewai.experimental.crew_agent_executor_flow.enforce_rpm_limit")
def test_call_llm_parser_error(
self, mock_enforce_rpm, mock_get_llm, mock_dependencies
):
"""Test call_llm_and_parse handles OutputParserError."""
from crewai.agents.parser import OutputParserError
mock_enforce_rpm.return_value = None
mock_get_llm.side_effect = OutputParserError("parse failed")
executor = CrewAgentExecutorFlow(**mock_dependencies)
result = executor.call_llm_and_parse()
assert result == "parser_error"
assert executor._last_parser_error is not None
@patch("crewai.experimental.crew_agent_executor_flow.get_llm_response")
@patch("crewai.experimental.crew_agent_executor_flow.enforce_rpm_limit")
@patch("crewai.experimental.crew_agent_executor_flow.is_context_length_exceeded")
def test_call_llm_context_error(
self,
mock_is_context_exceeded,
mock_enforce_rpm,
mock_get_llm,
mock_dependencies,
):
"""Test call_llm_and_parse handles context length error."""
mock_enforce_rpm.return_value = None
mock_get_llm.side_effect = Exception("context length")
mock_is_context_exceeded.return_value = True
executor = CrewAgentExecutorFlow(**mock_dependencies)
result = executor.call_llm_and_parse()
assert result == "context_error"
assert executor._last_context_error is not None
class TestFlowInvoke:
"""Test the invoke method that maintains backward compatibility."""
@pytest.fixture
def mock_dependencies(self):
"""Create mock dependencies."""
llm = Mock()
task = Mock()
task.description = "Test"
task.human_input = False
crew = Mock()
crew._short_term_memory = None
crew._long_term_memory = None
crew._entity_memory = None
crew._external_memory = None
agent = Mock()
agent.role = "Test"
agent.verbose = False
prompt = {"prompt": "Test {input} {tool_names} {tools}"}
return {
"llm": llm,
"task": task,
"crew": crew,
"agent": agent,
"prompt": prompt,
"max_iter": 10,
"tools": [],
"tools_names": "",
"stop_words": [],
"tools_description": "",
"tools_handler": Mock(),
}
@patch.object(CrewAgentExecutorFlow, "kickoff")
@patch.object(CrewAgentExecutorFlow, "_create_short_term_memory")
@patch.object(CrewAgentExecutorFlow, "_create_long_term_memory")
@patch.object(CrewAgentExecutorFlow, "_create_external_memory")
def test_invoke_success(
self,
mock_external_memory,
mock_long_term_memory,
mock_short_term_memory,
mock_kickoff,
mock_dependencies,
):
"""Test successful invoke without human feedback."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
# Mock kickoff to set the final answer in state
def mock_kickoff_side_effect():
executor.state.current_answer = AgentFinish(
thought="final thinking", output="Final result", text="complete"
)
mock_kickoff.side_effect = mock_kickoff_side_effect
inputs = {"input": "test", "tool_names": "", "tools": ""}
result = executor.invoke(inputs)
assert result == {"output": "Final result"}
mock_kickoff.assert_called_once()
mock_short_term_memory.assert_called_once()
mock_long_term_memory.assert_called_once()
mock_external_memory.assert_called_once()
@patch.object(CrewAgentExecutorFlow, "kickoff")
def test_invoke_failure_no_agent_finish(self, mock_kickoff, mock_dependencies):
"""Test invoke fails without AgentFinish."""
executor = CrewAgentExecutorFlow(**mock_dependencies)
executor.state.current_answer = AgentAction(
thought="thinking", tool="test", tool_input="test", text="action text"
)
inputs = {"input": "test", "tool_names": "", "tools": ""}
with pytest.raises(RuntimeError, match="without reaching a final answer"):
executor.invoke(inputs)
@patch.object(CrewAgentExecutorFlow, "kickoff")
@patch.object(CrewAgentExecutorFlow, "_create_short_term_memory")
@patch.object(CrewAgentExecutorFlow, "_create_long_term_memory")
@patch.object(CrewAgentExecutorFlow, "_create_external_memory")
def test_invoke_with_system_prompt(
self,
mock_external_memory,
mock_long_term_memory,
mock_short_term_memory,
mock_kickoff,
mock_dependencies,
):
"""Test invoke with system prompt configuration."""
mock_dependencies["prompt"] = {
"system": "System: {input}",
"user": "User: {input} {tool_names} {tools}",
}
executor = CrewAgentExecutorFlow(**mock_dependencies)
def mock_kickoff_side_effect():
executor.state.current_answer = AgentFinish(
thought="final thoughts", output="Done", text="complete"
)
mock_kickoff.side_effect = mock_kickoff_side_effect
inputs = {"input": "test", "tool_names": "", "tools": ""}
result = executor.invoke(inputs)
mock_short_term_memory.assert_called_once()
mock_long_term_memory.assert_called_once()
mock_external_memory.assert_called_once()
mock_kickoff.assert_called_once()
assert result == {"output": "Done"}
assert len(executor.state.messages) >= 2

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,401 @@
"""Unit tests for the @human_feedback decorator.
This module tests the @human_feedback decorator's validation logic,
async support, and attribute preservation functionality.
"""
from __future__ import annotations
import asyncio
from datetime import datetime
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from crewai.flow import Flow, human_feedback, listen, start
from crewai.flow.human_feedback import (
HumanFeedbackConfig,
HumanFeedbackResult,
)
class TestHumanFeedbackValidation:
"""Tests for decorator parameter validation."""
def test_emit_requires_llm(self):
"""Test that specifying emit without llm raises ValueError."""
with pytest.raises(ValueError) as exc_info:
@human_feedback(
message="Review this:",
emit=["approve", "reject"],
# llm not provided
)
def test_method(self):
return "output"
assert "llm is required" in str(exc_info.value)
def test_default_outcome_requires_emit(self):
"""Test that specifying default_outcome without emit raises ValueError."""
with pytest.raises(ValueError) as exc_info:
@human_feedback(
message="Review this:",
default_outcome="approve",
# emit not provided
)
def test_method(self):
return "output"
assert "requires emit" in str(exc_info.value)
def test_default_outcome_must_be_in_emit(self):
"""Test that default_outcome must be one of the emit values."""
with pytest.raises(ValueError) as exc_info:
@human_feedback(
message="Review this:",
emit=["approve", "reject"],
llm="gpt-4o-mini",
default_outcome="invalid_outcome",
)
def test_method(self):
return "output"
assert "must be one of" in str(exc_info.value)
def test_valid_configuration_with_routing(self):
"""Test that valid configuration with routing doesn't raise."""
@human_feedback(
message="Review this:",
emit=["approve", "reject"],
llm="gpt-4o-mini",
default_outcome="reject",
)
def test_method(self):
return "output"
# Should not raise
assert hasattr(test_method, "__human_feedback_config__")
assert test_method.__is_router__ is True
assert test_method.__router_paths__ == ["approve", "reject"]
def test_valid_configuration_without_routing(self):
"""Test that valid configuration without routing doesn't raise."""
@human_feedback(message="Review this:")
def test_method(self):
return "output"
# Should not raise
assert hasattr(test_method, "__human_feedback_config__")
assert not hasattr(test_method, "__is_router__") or not test_method.__is_router__
class TestHumanFeedbackConfig:
"""Tests for HumanFeedbackConfig dataclass."""
def test_config_creation(self):
"""Test HumanFeedbackConfig can be created with all parameters."""
config = HumanFeedbackConfig(
message="Test message",
emit=["a", "b"],
llm="gpt-4",
default_outcome="a",
metadata={"key": "value"},
)
assert config.message == "Test message"
assert config.emit == ["a", "b"]
assert config.llm == "gpt-4"
assert config.default_outcome == "a"
assert config.metadata == {"key": "value"}
class TestHumanFeedbackResult:
"""Tests for HumanFeedbackResult dataclass."""
def test_result_creation(self):
"""Test HumanFeedbackResult can be created with all fields."""
result = HumanFeedbackResult(
output={"title": "Test"},
feedback="Looks good",
outcome="approved",
method_name="test_method",
)
assert result.output == {"title": "Test"}
assert result.feedback == "Looks good"
assert result.outcome == "approved"
assert result.method_name == "test_method"
assert isinstance(result.timestamp, datetime)
assert result.metadata == {}
def test_result_with_metadata(self):
"""Test HumanFeedbackResult with custom metadata."""
result = HumanFeedbackResult(
output="test",
feedback="feedback",
metadata={"channel": "slack", "user": "test_user"},
)
assert result.metadata == {"channel": "slack", "user": "test_user"}
class TestDecoratorAttributePreservation:
"""Tests for preserving Flow decorator attributes."""
def test_preserves_start_method_attributes(self):
"""Test that @human_feedback preserves @start decorator attributes."""
class TestFlow(Flow):
@start()
@human_feedback(message="Review:")
def my_start_method(self):
return "output"
# Check that start method attributes are preserved
flow = TestFlow()
method = flow._methods.get("my_start_method")
assert method is not None
assert hasattr(method, "__is_start_method__") or "my_start_method" in flow._start_methods
def test_preserves_listen_method_attributes(self):
"""Test that @human_feedback preserves @listen decorator attributes."""
class TestFlow(Flow):
@start()
def begin(self):
return "start"
@listen("begin")
@human_feedback(message="Review:")
def review(self):
return "review output"
flow = TestFlow()
# The method should be registered as a listener
assert "review" in flow._listeners or any(
"review" in str(v) for v in flow._listeners.values()
)
def test_sets_router_attributes_when_emit_specified(self):
"""Test that router attributes are set when emit is specified."""
# Test the decorator directly without @start wrapping
@human_feedback(
message="Review:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def review_method(self):
return "output"
assert review_method.__is_router__ is True
assert review_method.__router_paths__ == ["approved", "rejected"]
class TestAsyncSupport:
"""Tests for async method support."""
def test_async_method_detection(self):
"""Test that async methods are properly detected and wrapped."""
@human_feedback(message="Review:")
async def async_method(self):
return "async output"
assert asyncio.iscoroutinefunction(async_method)
def test_sync_method_remains_sync(self):
"""Test that sync methods remain synchronous."""
@human_feedback(message="Review:")
def sync_method(self):
return "sync output"
assert not asyncio.iscoroutinefunction(sync_method)
class TestHumanFeedbackExecution:
"""Tests for actual human feedback execution."""
@patch("builtins.input", return_value="This looks great!")
@patch("builtins.print")
def test_basic_feedback_collection(self, mock_print, mock_input):
"""Test basic feedback collection without routing."""
class TestFlow(Flow):
@start()
@human_feedback(message="Please review:")
def generate(self):
return "Generated content"
flow = TestFlow()
with patch.object(flow, "_request_human_feedback", return_value="Great job!"):
result = flow.kickoff()
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.output == "Generated content"
assert flow.last_human_feedback.feedback == "Great job!"
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_empty_feedback_with_default_outcome(self, mock_print, mock_input):
"""Test empty feedback uses default_outcome."""
class TestFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approved", "needs_work"],
llm="gpt-4o-mini",
default_outcome="needs_work",
)
def review(self):
return "Content"
flow = TestFlow()
with patch.object(flow, "_request_human_feedback", return_value=""):
result = flow.kickoff()
assert result == "needs_work"
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.outcome == "needs_work"
@patch("builtins.input", return_value="Approved!")
@patch("builtins.print")
def test_feedback_collapsing(self, mock_print, mock_input):
"""Test that feedback is collapsed to an outcome."""
class TestFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def review(self):
return "Content"
flow = TestFlow()
with (
patch.object(flow, "_request_human_feedback", return_value="Looks great, approved!"),
patch.object(flow, "_collapse_to_outcome", return_value="approved"),
):
result = flow.kickoff()
assert result == "approved"
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.outcome == "approved"
class TestHumanFeedbackHistory:
"""Tests for human feedback history tracking."""
@patch("builtins.input", return_value="feedback")
@patch("builtins.print")
def test_history_accumulates(self, mock_print, mock_input):
"""Test that multiple feedbacks are stored in history."""
class TestFlow(Flow):
@start()
@human_feedback(message="Review step 1:")
def step1(self):
return "Step 1 output"
@listen(step1)
@human_feedback(message="Review step 2:")
def step2(self, prev):
return "Step 2 output"
flow = TestFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
flow.kickoff()
# Both feedbacks should be in history
assert len(flow.human_feedback_history) == 2
assert flow.human_feedback_history[0].method_name == "step1"
assert flow.human_feedback_history[1].method_name == "step2"
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_human_feedback_property_returns_last(self, mock_print, mock_input):
"""Test that human_feedback property returns the last result."""
class TestFlow(Flow):
@start()
@human_feedback(message="Review:")
def generate(self):
return "output"
flow = TestFlow()
with patch.object(flow, "_request_human_feedback", return_value="last feedback"):
flow.kickoff()
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.feedback == "last feedback"
assert flow.last_human_feedback is flow.last_human_feedback
class TestCollapseToOutcome:
"""Tests for the _collapse_to_outcome method."""
def test_exact_match(self):
"""Test exact match returns the correct outcome."""
flow = Flow()
with patch("crewai.llm.LLM") as MockLLM:
mock_llm = MagicMock()
mock_llm.call.return_value = "approved"
MockLLM.return_value = mock_llm
result = flow._collapse_to_outcome(
feedback="I approve this",
outcomes=["approved", "rejected"],
llm="gpt-4o-mini",
)
assert result == "approved"
def test_partial_match(self):
"""Test partial match finds the outcome in the response."""
flow = Flow()
with patch("crewai.llm.LLM") as MockLLM:
mock_llm = MagicMock()
mock_llm.call.return_value = "The outcome is approved based on the feedback"
MockLLM.return_value = mock_llm
result = flow._collapse_to_outcome(
feedback="Looks good",
outcomes=["approved", "rejected"],
llm="gpt-4o-mini",
)
assert result == "approved"
def test_fallback_to_first(self):
"""Test that unmatched response falls back to first outcome."""
flow = Flow()
with patch("crewai.llm.LLM") as MockLLM:
mock_llm = MagicMock()
mock_llm.call.return_value = "something completely different"
MockLLM.return_value = mock_llm
result = flow._collapse_to_outcome(
feedback="Unclear feedback",
outcomes=["approved", "rejected"],
llm="gpt-4o-mini",
)
assert result == "approved" # First in list

View File

@@ -0,0 +1,428 @@
"""Integration tests for the @human_feedback decorator with Flow.
This module tests the integration of @human_feedback with @listen,
routing behavior, multi-step flows, and state management.
"""
from __future__ import annotations
import asyncio
from datetime import datetime
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from pydantic import BaseModel
from crewai.flow import Flow, HumanFeedbackResult, human_feedback, listen, start
from crewai.flow.flow import FlowState
class TestRoutingIntegration:
"""Tests for routing integration with @listen decorators."""
@patch("builtins.input", return_value="I approve")
@patch("builtins.print")
def test_routes_to_matching_listener(self, mock_print, mock_input):
"""Test that collapsed outcome routes to the matching @listen method."""
execution_order = []
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def generate(self):
execution_order.append("generate")
return "content"
@listen("approved")
def on_approved(self):
execution_order.append("on_approved")
return "published"
@listen("rejected")
def on_rejected(self):
execution_order.append("on_rejected")
return "discarded"
flow = ReviewFlow()
with (
patch.object(flow, "_request_human_feedback", return_value="Approved!"),
patch.object(flow, "_collapse_to_outcome", return_value="approved"),
):
result = flow.kickoff()
assert "generate" in execution_order
assert "on_approved" in execution_order
assert "on_rejected" not in execution_order
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_default_outcome_routes_correctly(self, mock_print, mock_input):
"""Test that default_outcome routes when no feedback provided."""
executed_listener = []
class ReviewFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approved", "needs_work"],
llm="gpt-4o-mini",
default_outcome="needs_work",
)
def generate(self):
return "content"
@listen("approved")
def on_approved(self):
executed_listener.append("approved")
@listen("needs_work")
def on_needs_work(self):
executed_listener.append("needs_work")
flow = ReviewFlow()
with patch.object(flow, "_request_human_feedback", return_value=""):
flow.kickoff()
assert "needs_work" in executed_listener
assert "approved" not in executed_listener
class TestMultiStepFlows:
"""Tests for multi-step flows with multiple @human_feedback decorators."""
@patch("builtins.input", side_effect=["Good draft", "Final approved"])
@patch("builtins.print")
def test_multiple_feedback_steps(self, mock_print, mock_input):
"""Test a flow with multiple human feedback steps."""
class MultiStepFlow(Flow):
@start()
@human_feedback(message="Review draft:")
def draft(self):
return "Draft content"
@listen(draft)
@human_feedback(message="Final review:")
def final_review(self, prev_result: HumanFeedbackResult):
return f"Final content based on: {prev_result.feedback}"
flow = MultiStepFlow()
with patch.object(
flow, "_request_human_feedback", side_effect=["Good draft", "Approved"]
):
flow.kickoff()
# Both feedbacks should be recorded
assert len(flow.human_feedback_history) == 2
assert flow.human_feedback_history[0].method_name == "draft"
assert flow.human_feedback_history[0].feedback == "Good draft"
assert flow.human_feedback_history[1].method_name == "final_review"
assert flow.human_feedback_history[1].feedback == "Approved"
@patch("builtins.input", return_value="feedback")
@patch("builtins.print")
def test_mixed_feedback_and_regular_methods(self, mock_print, mock_input):
"""Test flow with both @human_feedback and regular methods."""
execution_order = []
class MixedFlow(Flow):
@start()
def generate(self):
execution_order.append("generate")
return "generated"
@listen(generate)
@human_feedback(message="Review:")
def review(self):
execution_order.append("review")
return "reviewed"
@listen(review)
def finalize(self, result):
execution_order.append("finalize")
return "finalized"
flow = MixedFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
flow.kickoff()
assert execution_order == ["generate", "review", "finalize"]
class TestStateManagement:
"""Tests for state management with human feedback."""
@patch("builtins.input", return_value="approved")
@patch("builtins.print")
def test_feedback_available_in_listener(self, mock_print, mock_input):
"""Test that feedback is accessible in downstream listeners."""
captured_feedback = []
class StateFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def review(self):
return "Content to review"
@listen("approved")
def on_approved(self):
# Access the feedback via property
captured_feedback.append(self.last_human_feedback)
return "done"
flow = StateFlow()
with (
patch.object(flow, "_request_human_feedback", return_value="Great content!"),
patch.object(flow, "_collapse_to_outcome", return_value="approved"),
):
flow.kickoff()
assert len(captured_feedback) == 1
result = captured_feedback[0]
assert isinstance(result, HumanFeedbackResult)
assert result.output == "Content to review"
assert result.feedback == "Great content!"
assert result.outcome == "approved"
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_history_preserved_across_steps(self, mock_print, mock_input):
"""Test that feedback history is preserved across flow execution."""
class HistoryFlow(Flow):
@start()
@human_feedback(message="Step 1:")
def step1(self):
return "Step 1"
@listen(step1)
@human_feedback(message="Step 2:")
def step2(self, result):
return "Step 2"
@listen(step2)
def final(self, result):
# Access history
return len(self.human_feedback_history)
flow = HistoryFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
result = flow.kickoff()
# Final method should see 2 feedback entries
assert result == 2
class TestAsyncFlowIntegration:
"""Tests for async flow integration."""
@pytest.mark.asyncio
async def test_async_flow_with_human_feedback(self):
"""Test that @human_feedback works with async flows."""
executed = []
class AsyncFlow(Flow):
@start()
@human_feedback(message="Review:")
async def async_review(self):
executed.append("async_review")
await asyncio.sleep(0.01) # Simulate async work
return "async content"
flow = AsyncFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
await flow.kickoff_async()
assert "async_review" in executed
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.output == "async content"
class TestWithStructuredState:
"""Tests for flows with structured (Pydantic) state."""
@patch("builtins.input", return_value="approved")
@patch("builtins.print")
def test_with_pydantic_state(self, mock_print, mock_input):
"""Test human feedback with structured Pydantic state."""
class ReviewState(FlowState):
content: str = ""
review_count: int = 0
class StructuredFlow(Flow[ReviewState]):
initial_state = ReviewState
@start()
@human_feedback(
message="Review:",
emit=["approved", "rejected"],
llm="gpt-4o-mini",
)
def review(self):
self.state.content = "Generated content"
self.state.review_count += 1
return self.state.content
@listen("approved")
def on_approved(self):
return f"Approved: {self.state.content}"
flow = StructuredFlow()
with (
patch.object(flow, "_request_human_feedback", return_value="LGTM"),
patch.object(flow, "_collapse_to_outcome", return_value="approved"),
):
result = flow.kickoff()
assert flow.state.review_count == 1
assert flow.last_human_feedback is not None
assert flow.last_human_feedback.feedback == "LGTM"
class TestMetadataPassthrough:
"""Tests for metadata passthrough functionality."""
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_metadata_included_in_result(self, mock_print, mock_input):
"""Test that metadata is passed through to HumanFeedbackResult."""
class MetadataFlow(Flow):
@start()
@human_feedback(
message="Review:",
metadata={"channel": "slack", "priority": "high"},
)
def review(self):
return "content"
flow = MetadataFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
flow.kickoff()
result = flow.last_human_feedback
assert result is not None
assert result.metadata == {"channel": "slack", "priority": "high"}
class TestEventEmission:
"""Tests for event emission during human feedback."""
@patch("builtins.input", return_value="test feedback")
@patch("builtins.print")
def test_events_emitted_on_feedback_request(self, mock_print, mock_input):
"""Test that events are emitted when feedback is requested."""
from crewai.events.event_listener import event_listener
class EventFlow(Flow):
@start()
@human_feedback(message="Review:")
def review(self):
return "content"
flow = EventFlow()
# We can't easily capture events in tests, but we can verify
# the flow executes without errors
with (
patch.object(
event_listener.formatter, "pause_live_updates", return_value=None
),
patch.object(
event_listener.formatter, "resume_live_updates", return_value=None
),
):
flow.kickoff()
assert flow.last_human_feedback is not None
class TestEdgeCases:
"""Tests for edge cases and error handling."""
@patch("builtins.input", return_value="")
@patch("builtins.print")
def test_empty_feedback_first_outcome_fallback(self, mock_print, mock_input):
"""Test that empty feedback without default uses first outcome."""
class FallbackFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["first", "second", "third"],
llm="gpt-4o-mini",
# No default_outcome specified
)
def review(self):
return "content"
flow = FallbackFlow()
with patch.object(flow, "_request_human_feedback", return_value=""):
result = flow.kickoff()
assert result == "first" # Falls back to first outcome
@patch("builtins.input", return_value="whitespace only ")
@patch("builtins.print")
def test_whitespace_only_feedback_treated_as_empty(self, mock_print, mock_input):
"""Test that whitespace-only feedback is treated as empty."""
class WhitespaceFlow(Flow):
@start()
@human_feedback(
message="Review:",
emit=["approve", "reject"],
llm="gpt-4o-mini",
default_outcome="reject",
)
def review(self):
return "content"
flow = WhitespaceFlow()
with patch.object(flow, "_request_human_feedback", return_value=" "):
result = flow.kickoff()
assert result == "reject" # Uses default because feedback is empty after strip
@patch("builtins.input", return_value="feedback")
@patch("builtins.print")
def test_feedback_result_without_routing(self, mock_print, mock_input):
"""Test that HumanFeedbackResult is returned when not routing."""
class NoRoutingFlow(Flow):
@start()
@human_feedback(message="Review:")
def review(self):
return "content"
flow = NoRoutingFlow()
with patch.object(flow, "_request_human_feedback", return_value="feedback"):
result = flow.kickoff()
# Result should be HumanFeedbackResult when not routing
assert isinstance(result, HumanFeedbackResult)
assert result.output == "content"
assert result.feedback == "feedback"
assert result.outcome is None # No routing, no outcome

978
uv.lock generated

File diff suppressed because it is too large Load Diff