diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md new file mode 100644 index 000000000..012a1c0bb --- /dev/null +++ b/.claude/CLAUDE.md @@ -0,0 +1,47 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository Overview + +This is a Claude Code Skills repository containing custom skills that extend Claude Code's capabilities. Skills are knowledge modules that Claude Code can invoke when specific conditions are met. + +## Skill Structure + +Each skill lives in `skills//` with this structure: +``` +skills// +├── SKILL.md # Required: frontmatter (name, description) + knowledge content +└── references/ # Optional: supplementary reference files + └── *.md +``` + +### SKILL.md Format + +```markdown +--- +name: skill-name +description: | + Multi-line description of when to use this skill. + Include specific trigger conditions. +--- + +# Main content follows... +``` + +The `description` field is critical - it tells Claude Code **when** to activate this skill. Be specific about trigger conditions. + +## Writing Effective Skills + +1. **Trigger Description**: Write clear, specific conditions in the YAML `description` field +2. **Actionable Content**: The markdown body should be directly usable guidance, not abstract theory +3. **Code Examples**: Include working code patterns that can be adapted +4. **Reference Files**: Split detailed reference material into `references/*.md` and link from SKILL.md + +## Current Skills + +- **crewai-architect**: Flow-first design patterns for CrewAI applications +- **crewai-enterprise-endpoint-manager**: REST API integration for deployed CrewAI crews/flows +- **crewai-tool-creator**: Custom tool development following best practices +- **software-architect**: Clean code with SOLID principles +- **streamlit**: Building interactive Python web apps diff --git a/.claude/README.md b/.claude/README.md new file mode 100644 index 000000000..726e4f804 --- /dev/null +++ b/.claude/README.md @@ -0,0 +1,103 @@ +# Claude Code Skills + +A collection of custom skills that extend Claude Code's capabilities with specialized domain knowledge. + +## What Are Skills? + +Skills are knowledge modules that Claude Code automatically activates when working on specific tasks. Each skill contains: + +- **Trigger conditions** - When to activate (defined in YAML frontmatter) +- **Expert knowledge** - Patterns, best practices, and code examples +- **Reference materials** - Deep-dive documentation for complex topics + +## Available Skills + +| Skill | Description | +|-------|-------------| +| **[crewai-architect](skills/crewai-architect/)** | Flow-first design patterns for CrewAI applications. Covers direct LLM calls, single agents, and crews within flows. | +| **[crewai-enterprise-endpoint-manager](skills/crewai-enterprise-endpoint-manager/)** | REST API integration for deployed CrewAI crews/flows in Enterprise (AOP). Authentication, monitoring, and result retrieval. | +| **[crewai-tool-creator](skills/crewai-tool-creator/)** | Custom tool development following CrewAI and Anthropic best practices. Input schemas, error handling, caching. | +| **[software-architect](skills/software-architect/)** | Clean code with SOLID principles. Single responsibility, dependency injection, interface segregation. | +| **[streamlit](skills/streamlit/)** | Building interactive Python web apps. Widgets, layouts, caching, session state, multipage apps. | + +## Using These Skills + +### Option 1: Clone to Claude Code Skills Directory + +```bash +# Clone into Claude Code's skills directory +git clone https://github.com/YOUR_USERNAME/skills.git ~/.claude/skills/my-skills +``` + +Skills are automatically discovered and activated based on their trigger descriptions. + +### Option 2: Reference in Project + +Add to your project's `.claude/settings.json`: + +```json +{ + "skills": { + "paths": ["/path/to/this/repo/skills"] + } +} +``` + +## Skill Structure + +``` +skills// +├── SKILL.md # Required: frontmatter + knowledge content +└── references/ # Optional: supplementary deep-dives + └── *.md +``` + +### SKILL.md Format + +```markdown +--- +name: skill-name +description: | + When to activate this skill. + Be specific about trigger conditions. +--- + +# Main Content + +Actionable guidance, patterns, and code examples. +``` + +## Contributing a New Skill + +1. Create a folder: `skills//` +2. Add `SKILL.md` with: + - YAML frontmatter (`name`, `description`) + - Practical, actionable content + - Working code examples +3. Optionally add `references/*.md` for detailed documentation +4. Submit a PR + +### Tips for Effective Skills + +- **Trigger Description**: Be specific about *when* the skill should activate +- **Actionable Content**: Include patterns and examples, not just theory +- **Code Examples**: Provide copy-paste-ready snippets +- **Reference Files**: Split detailed material into separate files + +## Repository Structure + +``` +. +├── CLAUDE.md # Instructions for Claude Code +├── README.md # This file +└── skills/ # Skill modules + ├── crewai-architect/ + ├── crewai-enterprise-endpoint-manager/ + ├── crewai-tool-creator/ + ├── software-architect/ + └── streamlit/ +``` + +## License + +MIT diff --git a/.claude/skills/crewai-architect/SKILL.md b/.claude/skills/crewai-architect/SKILL.md new file mode 100644 index 000000000..764a119dd --- /dev/null +++ b/.claude/skills/crewai-architect/SKILL.md @@ -0,0 +1,461 @@ +--- +name: crewai-architect +description: | + Guide for architecting CrewAI applications with mastery of different execution patterns. + Use this skill when: (1) Designing new CrewAI projects, (2) Choosing between direct LLM calls vs single agents vs crews within flows, + (3) Implementing flow-based orchestration with @start/@listen/@router decorators, (4) Embedding single agents or LLM calls within flow methods, + (5) Adding crews to flows with `crewai flow add-crew`, (6) Choosing kickoff methods (kickoff, kickoff_async, akickoff), + (7) Implementing structured outputs with Pydantic. Always start with Flows and add agency incrementally. +--- + +# CrewAI Architecture Guide + +**Core Principle: Start with Flows, add agency as needed.** + +Flows provide deterministic orchestration. Within flow methods, add agency incrementally: +1. Direct LLM calls (simplest) +2. Single agents (when tools needed) +3. Crews (when collaboration needed) + +## Architecture Decision Framework + +**Always start with a Flow.** Then choose the right level of agency for each step: + +| Agency Level | Within Flow Method | When to Use | +|--------------|-------------------|-------------| +| **Direct LLM** | `llm.call(messages)` | Structured extraction, no tools | +| **Single Agent** | `Agent(...).kickoff()` | Tool usage, single perspective | +| **Crew** | `MyCrew().crew().kickoff()` | Multi-agent collaboration | + +### Decision Tree for Each Flow Step + +``` +What does this step need? +├── Simple structured response → Direct LLM call +├── Tool usage, single perspective → Single Agent (inline) +└── Multi-perspective reasoning → Crew (use add-crew command) +``` + +## Multi-Agent Pattern Selection + +For complex applications requiring multiple agents working together, choose the appropriate orchestration pattern: + +| Pattern | Best For | Key Construct | +|---------|----------|---------------| +| **Generator-Critic** | Quality assurance with validation loops | Flow + @router + "revise" loop | +| **Iterative Refinement** | Progressive improvement to threshold | Flow + iteration counter | +| **Orchestrator-Worker** | Dynamic task decomposition | Flow + dynamic Agent() + asyncio.gather | +| **Task Guardrails** | Output validation with auto-retry | Task(guardrail=func, guardrail_max_retries=N) | +| **State Persistence** | Crash recovery, HITL resume | @persist() on Flow class | +| **HITL Webhooks** | Enterprise approval workflows | humanInputWebhook + /resume API | +| **Custom Manager** | Coordinated hierarchical delegation | Crew(manager_agent=..., process=hierarchical) | +| **Composite** | Enterprise production systems | Multiple patterns combined | + +### Quick Pattern Selection + +``` +Output quality critical? → Generator-Critic or Task Guardrails +Iterative improvement needed? → Iterative Refinement +Unknown task complexity? → Orchestrator-Worker (Dynamic Spawning) +Independent parallel tasks? → Parallel Fan-Out with asyncio.gather +Crash recovery needed? → State Persistence with @persist() +Human approval required? → HITL Webhooks +Enterprise production? → Composite (combine patterns) +``` + +See [references/multi-agent-patterns.md](references/multi-agent-patterns.md) for complete implementations. + +--- + +## Pattern 1: Flow with Direct LLM Calls + +**Use when:** Structured extraction, classification, simple transformations. + +```python +from crewai.flow.flow import Flow, start, listen +from crewai import LLM +from pydantic import BaseModel + +class TaskClassification(BaseModel): + category: str + priority: int + confidence: float + +class PipelineState(BaseModel): + input: str = "" + classification: TaskClassification | None = None + +class ClassificationFlow(Flow[PipelineState]): + def __init__(self): + super().__init__() + self.llm = LLM(model="gpt-4o") + + @start() + def classify_input(self): + llm = LLM(model="gpt-4o", response_format=TaskClassification) + result = llm.call(messages=[ + {"role": "user", "content": f"Classify: {self.state.input}"} + ]) + self.state.classification = result # Returns typed model directly + return result + +# Execute +flow = ClassificationFlow() +flow.state.input = "Urgent bug in production" +result = flow.kickoff() +``` + +## Pattern 2: Flow with Single Agents + +**Use when:** Step requires tools, memory, or multi-step reasoning from one perspective. + +**Define agents directly in flow methods:** + +```python +from crewai.flow.flow import Flow, start, listen, router +from crewai import Agent, LLM +from pydantic import BaseModel + +class AnalysisState(BaseModel): + data: str = "" + analysis: str = "" + needs_deep_dive: bool = False + +class AnalysisFlow(Flow[AnalysisState]): + + @start() + def quick_scan(self): + # Single agent defined inline for tool usage + scanner = Agent( + role="Data Scanner", + goal="Quickly scan data for anomalies", + backstory="Expert at rapid data assessment", + tools=[DataScanTool()], + llm=LLM(model="gpt-4o") + ) + result = scanner.kickoff(f"Scan: {self.state.data}") + self.state.needs_deep_dive = "anomaly" in result.raw.lower() + return result + + @router(quick_scan) + def route_analysis(self): + return "deep_dive" if self.state.needs_deep_dive else "summary" + + @listen("deep_dive") + def detailed_analysis(self): + # Another inline agent for different task + analyst = Agent( + role="Deep Analyst", + goal="Conduct thorough analysis", + backstory="Meticulous investigator", + tools=[AnalysisTool(), ChartTool()] + ) + result = analyst.kickoff(f"Deep dive: {self.state.data}") + self.state.analysis = result.raw + return result + + @listen("summary") + def quick_summary(self): + # Direct LLM call when no tools needed + llm = LLM(model="gpt-4o") + result = llm.call([{"role": "user", "content": f"Summarize: {self.state.data}"}]) + self.state.analysis = result + return result +``` + +**Async agent execution:** +```python +@listen(some_method) +async def async_analysis(self, data): + agent = Agent(role="Analyst", ...) + result = await agent.kickoff_async(f"Analyze: {data}") + return result +``` + +## Pattern 3: Flow with Crews + +**Use when:** Step requires multi-agent collaboration and autonomous problem-solving. + +### Adding a Crew to a Flow + +Use the CLI command: +```bash +crewai flow add-crew research_crew +``` + +This creates the crew structure under `src/your_project/crews/research_crew/`: +``` +research_crew/ +├── __init__.py +├── research_crew.py +└── config/ + ├── agents.yaml + └── tasks.yaml +``` + +### Using Crews in Flow Methods + +```python +from crewai.flow.flow import Flow, start, listen, router, or_ +from pydantic import BaseModel +from .crews.research_crew.research_crew import ResearchCrew +from .crews.writing_crew.writing_crew import WritingCrew + +class ContentState(BaseModel): + topic: str = "" + research: str = "" + article: str = "" + confidence: float = 0.0 + +class ContentPipeline(Flow[ContentState]): + + @start() + def validate_topic(self): + # Quick LLM validation + llm = LLM(model="gpt-4o", response_format=TopicValidation) + return llm.call([{"role": "user", "content": f"Validate topic: {self.state.topic}"}]) + + @listen(validate_topic) + def research_topic(self, validation): + # Crew for complex research + crew = ResearchCrew().crew() + result = crew.kickoff(inputs={"topic": self.state.topic}) + self.state.research = result.raw + self.state.confidence = 0.85 # From crew output + return result + + @router(research_topic) + def route_by_confidence(self): + if self.state.confidence > 0.8: + return "write_article" + return "needs_more_research" + + @listen("write_article") + def write_content(self): + # Another crew for writing + crew = WritingCrew().crew() + result = crew.kickoff(inputs={ + "topic": self.state.topic, + "research": self.state.research + }) + self.state.article = result.raw + return result + + @listen("needs_more_research") + def request_human_input(self): + return "Research inconclusive - human review needed" +``` + +## Flow Decorators Reference + +| Decorator | Purpose | Example | +|-----------|---------|---------| +| `@start()` | Entry point (multiple allowed, run parallel) | `@start()` | +| `@listen(method)` | Trigger on method completion | `@listen(validate)` | +| `@listen("label")` | Trigger on router label | `@listen("approved")` | +| `@router(method)` | Conditional routing, returns label | Returns `"approved"` or `"rejected"` | +| `and_(a, b)` | Trigger when ALL complete | `@listen(and_(task_a, task_b))` | +| `or_(a, b)` | Trigger when ANY completes | `@listen(or_("pass", "fail"))` | + +### Parallel Starts + +```python +class ParallelFlow(Flow[State]): + @start() # Runs in parallel + def fetch_data_a(self): + return "Data A" + + @start() # Runs in parallel + def fetch_data_b(self): + return "Data B" + + @listen(and_(fetch_data_a, fetch_data_b)) + def combine_results(self, result_a, result_b): + return f"{result_a} + {result_b}" +``` + +### Conditional Routing + +```python +@router(process_step) +def decide_path(self): + if self.state.score > 90: + return "excellent" + elif self.state.score > 70: + return "good" + return "needs_improvement" + +@listen("excellent") +def handle_excellent(self): ... + +@listen(or_("good", "needs_improvement")) +def handle_other(self): ... +``` + +## Kickoff Methods Reference + +### Crew Kickoff (within Flow methods) + +| Method | Use Case | +|--------|----------| +| `crew.kickoff(inputs={})` | Standard synchronous | +| `await crew.kickoff_async(inputs={})` | Thread-based async | +| `await crew.akickoff(inputs={})` | Native async (preferred) | +| `crew.kickoff_for_each(inputs=[...])` | Sequential batch | +| `await crew.akickoff_for_each([...])` | Concurrent batch (preferred) | + +### Flow Kickoff + +```python +flow = MyFlow() +flow.state.input = "data" # Set state +result = flow.kickoff() +print(flow.state.result) # Access final state +``` + +## Structured Output Patterns + +### At LLM Level (Direct calls) + +```python +from pydantic import BaseModel, Field + +class Analysis(BaseModel): + summary: str + key_points: list[str] + confidence: float = Field(ge=0, le=1) + +@start() +def analyze(self): + llm = LLM(model="gpt-4o", response_format=Analysis) + result = llm.call([...]) + return result # Returns typed Analysis instance directly +``` + +### At Agent Level + +```python +@listen(previous_step) +def agent_analysis(self, data): + agent = Agent(role="Analyst", ...) + result = agent.kickoff( + f"Analyze: {data}", + response_format=Analysis + ) + return result.pydantic +``` + +### At Task Level (Crew) + +```python +# In crew task definition +task = Task( + description="Generate report", + expected_output="Structured report", + agent=analyst, + output_pydantic=Report # Enforces structure +) + +# Access in flow +result = crew.kickoff(inputs={...}) +report = result.pydantic # Typed Report +``` + +## Complete Example: Adaptive Research Pipeline + +```python +from crewai.flow.flow import Flow, start, listen, router, or_ +from crewai import Agent, LLM +from pydantic import BaseModel +from .crews.research_crew.research_crew import ResearchCrew + +class ResearchState(BaseModel): + query: str = "" + complexity: str = "simple" + findings: str = "" + confidence: float = 0.0 + +class AdaptiveResearch(Flow[ResearchState]): + + @start() + def classify_query(self): + """LLM call for quick classification""" + llm = LLM(model="gpt-4o", response_format=QueryClassification) + result = llm.call([ + {"role": "user", "content": f"Classify complexity: {self.state.query}"} + ]) + self.state.complexity = result.complexity # LLM returns model directly + return result + + @router(classify_query) + def route_by_complexity(self): + return self.state.complexity # "simple", "moderate", or "complex" + + @listen("simple") + def quick_search(self): + """Single agent for simple queries""" + searcher = Agent( + role="Quick Researcher", + goal="Find answer efficiently", + tools=[SearchTool()], + llm=LLM(model="gpt-4o-mini") # Faster model + ) + result = searcher.kickoff(self.state.query) + self.state.findings = result.raw + self.state.confidence = 0.7 + return result + + @listen("moderate") + def standard_research(self): + """Single agent with more tools""" + researcher = Agent( + role="Researcher", + goal="Thorough research", + tools=[SearchTool(), AnalysisTool()], + llm=LLM(model="gpt-4o") + ) + result = researcher.kickoff(self.state.query) + self.state.findings = result.raw + self.state.confidence = 0.85 + return result + + @listen("complex") + def deep_research(self): + """Full crew for complex queries""" + crew = ResearchCrew().crew() + result = crew.kickoff(inputs={"query": self.state.query}) + self.state.findings = result.raw + self.state.confidence = 0.95 + return result + + @listen(or_("simple", "moderate", "complex")) + def finalize(self): + return { + "findings": self.state.findings, + "confidence": self.state.confidence + } + +# Execute +flow = AdaptiveResearch() +flow.state.query = "What are the implications of quantum computing on cryptography?" +result = flow.kickoff() +``` + +## Best Practices + +1. **Start with Flow** - Always use Flow as the orchestration layer +2. **Add agency incrementally** - LLM → Agent → Crew, only as needed +3. **Use `crewai flow add-crew`** - For creating crews within flows +4. **Define agents inline** - In flow methods for single-use agents +5. **Initialize LLMs in `__init__`** - Reuse LLM instances for efficiency +6. **Type your state** - Always use Pydantic BaseModel +7. **Use structured outputs** - `response_format` and `output_pydantic` +8. **Prefer native async** - `akickoff()` over `kickoff_async()` +9. **Route by state** - Use `@router` for conditional paths + +## Reference Files + +- [references/multi-agent-patterns.md](references/multi-agent-patterns.md) - **Multi-agent orchestration**: Generator-Critic, Iterative Refinement, Orchestrator-Worker, Task Guardrails, State Persistence, HITL Webhooks, Composite patterns +- [references/flow-patterns.md](references/flow-patterns.md) - Advanced flow patterns, HITL, resumable flows +- [references/crew-patterns.md](references/crew-patterns.md) - YAML config, process types, delegation +- [references/llm-patterns.md](references/llm-patterns.md) - Custom LLM integration, providers diff --git a/.claude/skills/crewai-architect/references/crew-patterns.md b/.claude/skills/crewai-architect/references/crew-patterns.md new file mode 100644 index 000000000..1227a400b --- /dev/null +++ b/.claude/skills/crewai-architect/references/crew-patterns.md @@ -0,0 +1,357 @@ +# Crew Patterns Reference + +## Table of Contents +- [YAML Configuration](#yaml-configuration) +- [Crew Class Structure](#crew-class-structure) +- [Process Types](#process-types) +- [Task Dependencies](#task-dependencies) +- [Agent Delegation](#agent-delegation) +- [Memory and Knowledge](#memory-and-knowledge) + +## YAML Configuration + +### Creating a Crew with CLI + +```bash +crewai flow add-crew research_crew +``` + +Creates: +``` +src/project/crews/research_crew/ +├── __init__.py +├── research_crew.py +└── config/ + ├── agents.yaml + └── tasks.yaml +``` + +### agents.yaml + +```yaml +researcher: + role: "Senior Research Analyst" + goal: "Conduct thorough research on {topic} and identify key insights" + backstory: | + You are a veteran researcher with 15 years of experience + in market analysis. You excel at finding hidden patterns + and connecting disparate data points. + verbose: true + allow_delegation: false + +analyst: + role: "Data Analyst" + goal: "Analyze research data and produce actionable recommendations" + backstory: | + Expert at transforming raw research into strategic insights. + Known for clear, data-driven conclusions. + verbose: true + allow_delegation: true +``` + +**Variable interpolation:** Use `{variable}` for dynamic values passed at kickoff. + +### tasks.yaml + +```yaml +research_task: + description: | + Research the following topic thoroughly: {topic} + + Focus on: + - Current market trends + - Key players and competitors + - Recent developments (last 6 months) + - Potential risks and opportunities + expected_output: | + Comprehensive research report with: + - Executive summary + - Detailed findings + - Data sources cited + agent: researcher + +analysis_task: + description: | + Analyze the research findings and provide recommendations. + Consider: {analysis_focus} + expected_output: | + Strategic analysis with: + - Key insights + - Recommendations (prioritized) + - Risk assessment + agent: analyst + context: + - research_task +``` + +## Crew Class Structure + +```python +from crewai import Agent, Crew, Process, Task +from crewai.project import CrewBase, agent, crew, task +from crewai.agents.agent_builder.base_agent import BaseAgent +from typing import List + +@CrewBase +class ResearchCrew: + """Research crew for thorough topic investigation.""" + + agents: List[BaseAgent] + tasks: List[Task] + agents_config = "config/agents.yaml" + tasks_config = "config/tasks.yaml" + + @agent + def researcher(self) -> Agent: + return Agent( + config=self.agents_config["researcher"], # type: ignore[index] + tools=[SearchTool(), WebScrapeTool()], + verbose=True, + ) + + @agent + def analyst(self) -> Agent: + return Agent( + config=self.agents_config["analyst"], # type: ignore[index] + tools=[AnalysisTool()], + verbose=True, + ) + + @task + def research_task(self) -> Task: + return Task( + config=self.tasks_config["research_task"], # type: ignore[index] + ) + + @task + def analysis_task(self) -> Task: + return Task( + config=self.tasks_config["analysis_task"], # type: ignore[index] + output_pydantic=AnalysisReport, # Structured output + ) + + @crew + def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.sequential, + verbose=True, + ) +``` + +**Critical:** Always use `# type: ignore[index]` for config access. + +## Process Types + +### Sequential (Default) + +Tasks execute in order, each receiving context from previous. + +```python +@crew +def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.sequential, + ) +``` + +### Hierarchical + +Manager agent coordinates work distribution. + +```python +from crewai import LLM + +@crew +def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.hierarchical, + manager_llm=LLM(model="gpt-4o"), # Required for hierarchical + ) +``` + +### Parallel Task Groups + +```python +@crew +def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.sequential, + parallel_task_execution=True, # Tasks without dependencies run parallel + ) +``` + +## Task Dependencies + +### Context Chaining + +```yaml +# tasks.yaml +gather_task: + description: "Gather initial data" + agent: gatherer + +analyze_task: + description: "Analyze gathered data" + agent: analyst + context: + - gather_task # Receives gather_task output + +report_task: + description: "Write final report" + agent: writer + context: + - gather_task # Has access to both + - analyze_task # previous outputs +``` + +### Programmatic Context + +```python +@task +def synthesis_task(self) -> Task: + return Task( + config=self.tasks_config["synthesis_task"], # type: ignore[index] + context=[self.research_task(), self.analysis_task()], + ) +``` + +## Agent Delegation + +### Enable Delegation + +```yaml +# agents.yaml +manager: + role: "Project Manager" + goal: "Coordinate team and delegate effectively" + allow_delegation: true # Can delegate to other agents + +specialist: + role: "Technical Specialist" + goal: "Handle technical implementation" + allow_delegation: false # Handles own work +``` + +### Delegation in Hierarchical + +```python +@crew +def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.hierarchical, + manager_llm=LLM(model="gpt-4o"), + manager_agent=self.manager(), # Optional custom manager + ) +``` + +## Memory and Knowledge + +### Enable Memory + +```python +@crew +def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.sequential, + memory=True, # Enable short-term memory + verbose=True, + ) +``` + +### Knowledge Sources + +```python +from crewai.knowledge.source.text_file_knowledge_source import TextFileKnowledgeSource + +@crew +def crew(self) -> Crew: + knowledge = TextFileKnowledgeSource( + file_paths=["docs/guidelines.md", "docs/reference.md"] + ) + return Crew( + agents=self.agents, + tasks=self.tasks, + knowledge_sources=[knowledge], + ) +``` + +## Output Patterns + +### Structured Task Output + +```python +from pydantic import BaseModel, Field + +class ResearchReport(BaseModel): + summary: str + findings: list[str] + sources: list[str] + confidence: float = Field(ge=0, le=1) + +@task +def research_task(self) -> Task: + return Task( + config=self.tasks_config["research_task"], # type: ignore[index] + output_pydantic=ResearchReport, + ) +``` + +### File Output + +```python +@task +def report_task(self) -> Task: + return Task( + config=self.tasks_config["report_task"], # type: ignore[index] + output_file="outputs/report.md", + ) +``` + +### Accessing Outputs + +```python +# In flow method +result = ResearchCrew().crew().kickoff(inputs={"topic": "AI"}) + +# Raw output +print(result.raw) + +# Structured output (if output_pydantic set) +report = result.pydantic +print(report.summary) + +# Dictionary access +print(result["summary"]) + +# JSON (if output_json set) +data = result.json +``` + +## Using Crew in Flow + +```python +from project_name.crews.research_crew.research_crew import ResearchCrew + +class ResearchFlow(Flow[State]): + @listen(validate_input) + def do_research(self, validated): + crew = ResearchCrew().crew() + result = crew.kickoff(inputs={ + "topic": self.state.topic, + "analysis_focus": self.state.focus + }) + self.state.research = result.pydantic + return result +``` diff --git a/.claude/skills/crewai-architect/references/flow-patterns.md b/.claude/skills/crewai-architect/references/flow-patterns.md new file mode 100644 index 000000000..7a8aa51ef --- /dev/null +++ b/.claude/skills/crewai-architect/references/flow-patterns.md @@ -0,0 +1,297 @@ +# Advanced Flow Patterns + +## Table of Contents +- [Resumable Flows](#resumable-flows) +- [Human-in-the-Loop (HITL)](#human-in-the-loop-hitl) +- [Parallel Execution](#parallel-execution) +- [Error Handling and Retries](#error-handling-and-retries) +- [State Persistence](#state-persistence) + +## Resumable Flows + +Flows can be resumed from specific points using conditional starts: + +```python +from crewai.flow.flow import Flow, start, listen + +class ResumableFlow(Flow[State]): + @start() # Unconditional start + def init(self): + self.state.initialized = True + return "initialized" + + @start("init") # Conditional: runs after init OR as external trigger + def maybe_begin(self): + return "began" + + @listen(and_(init, maybe_begin)) + def proceed(self): + return "proceeding" +``` + +### Resuming from External Trigger + +```python +# Start from beginning +flow = MyFlow() +result = flow.kickoff() + +# Resume from specific method +flow = MyFlow() +flow.state.previous_data = loaded_state +result = flow.kickoff(start_method="maybe_begin") +``` + +## Human-in-the-Loop (HITL) + +### Pause for Human Review + +```python +class HITLFlow(Flow[ReviewState]): + @start() + def generate_draft(self): + agent = Agent(role="Writer", ...) + result = agent.kickoff("Write initial draft") + self.state.draft = result.raw + return result + + @router(generate_draft) + def check_confidence(self): + if self.state.confidence < 0.7: + return "needs_human_review" + return "auto_approve" + + @listen("needs_human_review") + def pause_for_review(self): + # Save state for later resume + self.state.status = "awaiting_review" + self.state.save() # Persist state + return "Paused for human review" + + @listen("auto_approve") + def proceed_automatically(self): + return self.finalize() + + @start("human_approved") # Resume point after human approval + def after_human_review(self): + # Human has reviewed and approved + return self.finalize() + + def finalize(self): + return {"final": self.state.draft} +``` + +### Integration with External Systems + +```python +@listen("needs_approval") +def request_approval(self): + # Send to Slack, email, or queue + send_approval_request( + content=self.state.draft, + callback_id=self.state.flow_id + ) + self.state.status = "pending_approval" + return "Approval requested" +``` + +## Parallel Execution + +### Multiple Start Points + +```python +class ParallelDataFlow(Flow[DataState]): + @start() + def fetch_source_a(self): + return api_client.get_data_a() + + @start() + def fetch_source_b(self): + return api_client.get_data_b() + + @start() + def fetch_source_c(self): + return api_client.get_data_c() + + @listen(and_(fetch_source_a, fetch_source_b, fetch_source_c)) + def merge_all_sources(self, a, b, c): + self.state.merged = {**a, **b, **c} + return self.state.merged +``` + +### Async Within Methods + +```python +@listen(previous_step) +async def parallel_processing(self, data): + # Process multiple items concurrently + tasks = [ + self.process_item(item) + for item in data["items"] + ] + results = await asyncio.gather(*tasks) + return results + +async def process_item(self, item): + agent = Agent(role="Processor", ...) + return await agent.kickoff_async(f"Process: {item}") +``` + +### Crew Batch Processing + +```python +@listen(gather_inputs) +async def batch_analyze(self, inputs): + crew = AnalysisCrew().crew() + + # Concurrent batch processing + results = await crew.akickoff_for_each([ + {"item": item} for item in inputs + ]) + + self.state.analyses = [r.raw for r in results] + return results +``` + +## Error Handling and Retries + +### Try-Catch Pattern + +```python +class RobustFlow(Flow[State]): + @start() + def risky_operation(self): + try: + result = external_api.call() + self.state.result = result + return "success" + except APIError as e: + self.state.error = str(e) + self.state.retry_count = getattr(self.state, 'retry_count', 0) + 1 + return "error" + + @router(risky_operation) + def handle_result(self): + if hasattr(self.state, 'error'): + if self.state.retry_count < 3: + return "retry" + return "failed" + return "success" + + @listen("retry") + def retry_operation(self): + import time + time.sleep(2 ** self.state.retry_count) # Exponential backoff + return self.risky_operation() + + @listen("failed") + def handle_failure(self): + return {"error": self.state.error, "retries": self.state.retry_count} +``` + +### Graceful Degradation + +```python +@listen(process_step) +def with_fallback(self, primary_result): + if not primary_result or primary_result.get("status") == "failed": + # Fallback to simpler approach + llm = LLM(model="gpt-4o-mini") + return llm.call([{"role": "user", "content": "Simple fallback..."}]) + return primary_result +``` + +## State Persistence + +### Pydantic State with Validation + +```python +from pydantic import BaseModel, Field, field_validator +from typing import Optional +from datetime import datetime + +class PersistentState(BaseModel): + flow_id: str = Field(default_factory=lambda: str(uuid.uuid4())) + created_at: datetime = Field(default_factory=datetime.now) + status: str = "pending" + data: dict = {} + error: Optional[str] = None + retry_count: int = 0 + + @field_validator('status') + @classmethod + def validate_status(cls, v): + valid = ['pending', 'processing', 'completed', 'failed', 'paused'] + if v not in valid: + raise ValueError(f'Status must be one of {valid}') + return v + + def save(self): + with open(f"state_{self.flow_id}.json", "w") as f: + f.write(self.model_dump_json()) + + @classmethod + def load(cls, flow_id: str): + with open(f"state_{flow_id}.json") as f: + return cls.model_validate_json(f.read()) +``` + +### Using Persistent State + +```python +class PersistentFlow(Flow[PersistentState]): + @start() + def begin_processing(self): + self.state.status = "processing" + self.state.save() + return self.do_work() + + @listen(begin_processing) + def checkpoint(self, result): + self.state.data["checkpoint_1"] = result + self.state.save() + return result + +# Resume from saved state +saved_state = PersistentState.load("flow-123") +flow = PersistentFlow() +flow.state = saved_state +flow.kickoff(start_method="checkpoint") +``` + +## Flow Composition + +### Nested Flows + +```python +class SubFlow(Flow[SubState]): + @start() + def sub_process(self): + return "sub result" + +class MainFlow(Flow[MainState]): + @listen(setup) + def run_subflow(self): + sub = SubFlow() + sub.state.input = self.state.data + result = sub.kickoff() + self.state.sub_result = result + return result +``` + +### Flow Factory Pattern + +```python +def create_flow(flow_type: str) -> Flow: + flows = { + "analysis": AnalysisFlow, + "research": ResearchFlow, + "writing": WritingFlow + } + return flows[flow_type]() + +# Usage +flow = create_flow(config["flow_type"]) +flow.state.input = data +result = flow.kickoff() +``` diff --git a/.claude/skills/crewai-architect/references/llm-patterns.md b/.claude/skills/crewai-architect/references/llm-patterns.md new file mode 100644 index 000000000..577dd6023 --- /dev/null +++ b/.claude/skills/crewai-architect/references/llm-patterns.md @@ -0,0 +1,269 @@ +# LLM Integration Patterns + +## Table of Contents +- [Built-in LLM Configuration](#built-in-llm-configuration) +- [Provider-Specific Setup](#provider-specific-setup) +- [Custom LLM Implementation](#custom-llm-implementation) +- [Direct LLM Calls in Flows](#direct-llm-calls-in-flows) +- [Structured Responses](#structured-responses) + +## Built-in LLM Configuration + +### Basic LLM Class + +```python +from crewai import LLM + +# OpenAI +llm = LLM(model="gpt-4o", temperature=0.7) + +# Anthropic +llm = LLM(model="anthropic/claude-sonnet-4-20250514") + +# Google +llm = LLM(model="gemini/gemini-2.0-flash") + +# Local (Ollama) +llm = LLM(model="ollama/llama3.2") +``` + +### LLM Parameters + +```python +llm = LLM( + model="gpt-4o", + temperature=0.7, # Creativity (0-1) + max_tokens=4096, # Max response length + top_p=0.9, # Nucleus sampling + frequency_penalty=0.0, # Reduce repetition + presence_penalty=0.0, # Encourage new topics + seed=42, # Reproducibility + response_format=MyModel, # Pydantic model for structured output +) +``` + +## Provider-Specific Setup + +### OpenAI + +```bash +# .env +OPENAI_API_KEY=sk-... +``` + +```python +llm = LLM(model="gpt-4o") +# or +llm = LLM(model="gpt-4o-mini") # Faster, cheaper +``` + +### Anthropic + +```bash +# .env +ANTHROPIC_API_KEY=sk-ant-... +``` + +```python +llm = LLM(model="anthropic/claude-sonnet-4-20250514") +``` + +### Google Gemini + +```bash +# .env +GOOGLE_API_KEY=AIza... +``` + +```python +# Via LiteLLM +llm = LLM(model="gemini/gemini-2.0-flash") + +# Via OpenAI-compatible endpoint +llm = LLM( + model="openai/gemini-2.0-flash", + base_url="https://generativelanguage.googleapis.com/v1beta/openai/", + api_key="your-gemini-key" +) +``` + +### Azure OpenAI + +```bash +# .env +AZURE_API_KEY=... +AZURE_API_BASE=https://your-resource.openai.azure.com/ +AZURE_API_VERSION=2024-02-01 +``` + +```python +llm = LLM( + model="azure/your-deployment-name", + api_key=os.getenv("AZURE_API_KEY"), + base_url=os.getenv("AZURE_API_BASE"), +) +``` + +### Local Models (Ollama) + +```bash +# Start Ollama +ollama serve +ollama pull llama3.2 +``` + +```python +llm = LLM( + model="ollama/llama3.2", + base_url="http://localhost:11434" +) +``` + +## Direct LLM Calls in Flows + +### Simple Call + +```python +class MyFlow(Flow[State]): + def __init__(self): + super().__init__() + self.llm = LLM(model="gpt-4o") + + @start() + def process(self): + response = self.llm.call(messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": f"Process: {self.state.input}"} + ]) + return response +``` + +### With Message History + +```python +@listen(previous_step) +def conversation(self, context): + messages = [ + {"role": "system", "content": "Expert analyst."}, + {"role": "user", "content": "Initial question"}, + {"role": "assistant", "content": context}, # Previous response + {"role": "user", "content": "Follow-up question"} + ] + return self.llm.call(messages=messages) +``` + +## Structured Responses + +### With LLM Class + +```python +from pydantic import BaseModel, Field + +class Analysis(BaseModel): + summary: str = Field(description="Brief summary") + key_points: list[str] = Field(description="Main findings") + confidence: float = Field(ge=0, le=1, description="Confidence score") + recommendation: str + +@start() +def analyze(self): + llm = LLM(model="gpt-4o", response_format=Analysis) + result = llm.call(messages=[ + {"role": "user", "content": f"Analyze: {self.state.data}"} + ]) + self.state.analysis = result # LLM.call() returns model directly + return result +``` + +### Complex Nested Models + +```python +from pydantic import BaseModel +from typing import Optional + +class Finding(BaseModel): + title: str + description: str + severity: str # "low", "medium", "high" + +class RiskAssessment(BaseModel): + overall_risk: str + findings: list[Finding] + mitigations: list[str] + +class FullReport(BaseModel): + executive_summary: str + risk_assessment: RiskAssessment + recommendations: list[str] + next_steps: Optional[list[str]] = None + +llm = LLM(model="gpt-4o", response_format=FullReport) +result = llm.call(messages=[...]) +report = result # Returns fully typed FullReport directly +``` + +### Validation and Error Handling + +```python +@start() +def safe_extraction(self): + llm = LLM(model="gpt-4o", response_format=DataModel) + try: + result = llm.call(messages=[...]) + if result: # LLM returns model directly when response_format is set + self.state.data = result + return "success" + else: + self.state.error = "No structured output" + return "failed" + except Exception as e: + self.state.error = str(e) + return "failed" +``` + +## Model Selection Guide + +| Use Case | Recommended Model | Notes | +|----------|------------------|-------| +| Complex reasoning | gpt-4o, claude-sonnet-4-20250514 | Best quality | +| Fast responses | gpt-4o-mini, gemini-2.0-flash | Good balance | +| Cost-sensitive | gpt-4o-mini, ollama/llama3.2 | Lowest cost | +| Long context | gpt-4o (128k), claude (200k) | Large documents | +| Structured output | gpt-4o, gpt-4o-mini | Best JSON mode | +| Privacy-sensitive | ollama/*, local models | No data leaves | + +## Caching and Optimization + +### LLM Instance Reuse + +```python +class OptimizedFlow(Flow[State]): + def __init__(self): + super().__init__() + # Initialize once, reuse across methods + self.fast_llm = LLM(model="gpt-4o-mini") + self.smart_llm = LLM(model="gpt-4o") + + @start() + def quick_classify(self): + # Use fast model for simple tasks + return self.fast_llm.call([...]) + + @listen(quick_classify) + def deep_analysis(self, classification): + # Use smart model for complex tasks + return self.smart_llm.call([...]) +``` + +### Temperature by Task + +```python +# Classification: low temperature for consistency +classifier = LLM(model="gpt-4o", temperature=0.1) + +# Creative: higher temperature for variety +creative = LLM(model="gpt-4o", temperature=0.9) + +# Structured extraction: zero temperature +extractor = LLM(model="gpt-4o", temperature=0, response_format=DataModel) +``` diff --git a/.claude/skills/crewai-architect/references/multi-agent-patterns.md b/.claude/skills/crewai-architect/references/multi-agent-patterns.md new file mode 100644 index 000000000..46e3adb70 --- /dev/null +++ b/.claude/skills/crewai-architect/references/multi-agent-patterns.md @@ -0,0 +1,722 @@ +# Multi-Agent Design Patterns + +Advanced patterns for orchestrating multiple agents in complex applications. These patterns address common challenges: quality assurance, dynamic scaling, fault tolerance, and human oversight. + +## Pattern Selection Guide + +### Decision Tree + +``` +What is your primary concern? +├── Output Quality Critical → Generator-Critic or Task Guardrails +├── Iterative Improvement → Iterative Refinement +├── Unknown Task Complexity → Orchestrator-Worker (Dynamic Spawning) +├── Independent Parallel Tasks → Parallel Fan-Out +├── Coordinator Needed → Hierarchical with Custom Manager +├── Crash Recovery / Long Tasks → State Persistence +├── Human Approval Required → HITL Webhooks +└── Enterprise Production → Composite (combine patterns) +``` + +### Quick Reference + +| Pattern | Best For | Key CrewAI Construct | +|---------|----------|---------------------| +| **Generator-Critic** | Quality gates, validation loops | Flow + @router + "revise" loop | +| **Iterative Refinement** | Progressive improvement | Flow + iteration counter + exit condition | +| **Orchestrator-Worker** | Dynamic task decomposition | Flow + inline Agent() + asyncio.gather | +| **Parallel Fan-Out** | Concurrent independent tasks | asyncio.gather + akickoff() | +| **Task Guardrails** | Output validation | Task(guardrail=func, guardrail_max_retries=N) | +| **State Persistence** | Crash recovery, HITL | @persist() on Flow class | +| **HITL Webhooks** | Enterprise approval flows | humanInputWebhook + /resume API | +| **Custom Manager** | Coordinated delegation | Crew(manager_agent=..., process=Process.hierarchical) | +| **Composite** | Enterprise systems | Nested patterns combined | + +--- + +## Generator-Critic Pattern + +**Use when:** Output quality is critical and requires validation before acceptance (legal, medical, financial content). + +**Concept:** One agent/crew generates content, another critiques it. Based on quality score, either approve or loop back for revision. + +```python +from crewai.flow.flow import Flow, start, listen, router +from crewai import Crew +from pydantic import BaseModel + +class GeneratorCriticState(BaseModel): + content: str = "" + critique: str = "" + quality_score: float = 0.0 + iteration: int = 0 + max_iterations: int = 3 + +class GeneratorCriticFlow(Flow[GeneratorCriticState]): + + @start() + def generate(self): + """Generator crew creates content""" + crew = GeneratorCrew().crew() + result = crew.kickoff(inputs={"topic": self.state.topic}) + self.state.content = result.raw + self.state.iteration += 1 + return result + + @listen(generate) + def critique(self, generated): + """Critic crew evaluates quality""" + crew = CriticCrew().crew() + result = crew.kickoff(inputs={"content": self.state.content}) + self.state.critique = result.raw + self.state.quality_score = result.pydantic.score # Assumes structured output + return result + + @router(critique) + def check_quality(self): + if self.state.quality_score >= 0.8: + return "approved" + elif self.state.iteration >= self.state.max_iterations: + return "max_iterations" + return "revise" + + @listen("revise") + def revise_content(self): + """Feed critique back to generator""" + crew = GeneratorCrew().crew() + result = crew.kickoff(inputs={ + "topic": self.state.topic, + "previous_attempt": self.state.content, + "feedback": self.state.critique + }) + self.state.content = result.raw + self.state.iteration += 1 + return result + + @listen(revise_content) + def re_critique(self, revised): + """Loop back to critique""" + return self.critique(revised) + + @listen("approved") + def finalize(self): + return {"content": self.state.content, "iterations": self.state.iteration} + + @listen("max_iterations") + def handle_max(self): + return {"content": self.state.content, "warning": "Max iterations reached"} +``` + +**When NOT to use:** Simple tasks where quality is easily validated, or when iteration cost is too high. + +--- + +## Iterative Refinement Pattern + +**Use when:** Output requires progressive improvement toward a quality threshold (optimization, polishing). + +**Difference from Generator-Critic:** Focuses on continuous improvement rather than pass/fail validation. + +```python +from crewai.flow.flow import Flow, start, listen, router +from crewai import Agent, LLM +from pydantic import BaseModel + +class RefinementState(BaseModel): + draft: str = "" + quality_score: float = 0.0 + iteration: int = 0 + max_iterations: int = 5 + target_quality: float = 0.9 + +class IterativeRefinementFlow(Flow[RefinementState]): + + @start() + def create_initial(self): + """Create initial draft""" + writer = Agent( + role="Content Writer", + goal="Create high-quality initial draft", + llm=LLM(model="gpt-4o") + ) + result = writer.kickoff(f"Write about: {self.state.topic}") + self.state.draft = result.raw + return result + + @listen(create_initial) + def assess_and_refine(self, draft): + """Assess quality and refine if needed""" + self.state.iteration += 1 + + # Assess quality + llm = LLM(model="gpt-4o", response_format=QualityAssessment) + assessment = llm.call([ + {"role": "user", "content": f"Score 0-1 and suggest improvements:\n\n{self.state.draft}"} + ]) + self.state.quality_score = assessment.score + + if self.state.quality_score >= self.state.target_quality: + return "converged" + if self.state.iteration >= self.state.max_iterations: + return "max_reached" + + # Refine + refiner = Agent(role="Editor", goal="Improve based on feedback", llm=LLM(model="gpt-4o")) + result = refiner.kickoff(f"Improve:\n{self.state.draft}\n\nFeedback:\n{assessment.improvements}") + self.state.draft = result.raw + return "continue" + + @router(assess_and_refine) + def route(self): + # Router returns the string from assess_and_refine + pass # The return from assess_and_refine is the route + + @listen("continue") + def continue_refinement(self): + """Loop back for another iteration""" + return self.assess_and_refine(self.state.draft) + + @listen("converged") + def output_final(self): + return {"content": self.state.draft, "iterations": self.state.iteration} +``` + +--- + +## Orchestrator-Worker Pattern (Dynamic Spawning) + +**Use when:** Task complexity is unknown upfront and requires dynamic scaling of subagents. + +**Concept:** Lead agent analyzes complexity, spawns appropriate number of specialized workers, then synthesizes results. + +```python +from crewai.flow.flow import Flow, start, listen, router +from crewai import Agent, LLM +from pydantic import BaseModel +import asyncio + +class OrchestratorState(BaseModel): + query: str = "" + complexity: str = "moderate" + worker_count: int = 1 + worker_results: list[str] = [] + synthesis: str = "" + +class OrchestratorWorkerFlow(Flow[OrchestratorState]): + + @start() + def analyze_complexity(self): + """Orchestrator analyzes task and plans""" + llm = LLM(model="gpt-4o", response_format=TaskPlan) + plan = llm.call([ + {"role": "system", "content": "Analyze query complexity. Return subtasks."}, + {"role": "user", "content": f"Query: {self.state.query}"} + ]) + + self.state.complexity = plan.complexity + # Scale workers to complexity: simple=1, moderate=3, complex=5 + complexity_map = {"simple": 1, "moderate": 3, "complex": 5} + self.state.worker_count = complexity_map.get(plan.complexity, 3) + return plan + + @listen(analyze_complexity) + async def spawn_workers(self, plan): + """Dynamically create and run worker agents in parallel""" + worker_tasks = [] + + for i, subtask in enumerate(plan.subtasks[:self.state.worker_count]): + # Create fresh agent per subtask (clean context) + agent = Agent( + role=f"Research Specialist #{i+1}", + goal=subtask.objective, + backstory=f"Expert in {subtask.domain}", + tools=[SearchTool()], + llm=LLM(model="gpt-4o-mini") # Cheaper model for workers + ) + worker_tasks.append(agent.akickoff(subtask.instructions)) + + # Run all workers concurrently + results = await asyncio.gather(*worker_tasks) + self.state.worker_results = [r.raw for r in results] + return results + + @listen(spawn_workers) + def synthesize(self, worker_outputs): + """Orchestrator synthesizes all findings""" + synthesizer = Agent( + role="Lead Researcher", + goal="Synthesize findings into coherent answer", + llm=LLM(model="gpt-4o") # Best model for synthesis + ) + + combined = "\n\n---\n\n".join([ + f"Worker {i+1}:\n{result}" for i, result in enumerate(self.state.worker_results) + ]) + + result = synthesizer.kickoff( + f"Query: {self.state.query}\n\nFindings:\n{combined}\n\nSynthesize into answer." + ) + self.state.synthesis = result.raw + return result +``` + +**Scaling guidance:** +- Simple queries: 1 worker, 3-10 tool calls +- Moderate: 3 workers in parallel +- Complex: 5+ workers, may need multiple rounds + +--- + +## Parallel Fan-Out Pattern + +**Use when:** Multiple independent subtasks can run concurrently for speed. + +```python +from crewai.flow.flow import Flow, start, listen, and_ +from crewai import Crew +import asyncio + +class ParallelState(BaseModel): + data: dict = {} + security_result: str = "" + performance_result: str = "" + style_result: str = "" + final_report: str = "" + +class ParallelAnalysisFlow(Flow[ParallelState]): + + @start() + async def fan_out_analysis(self): + """Run multiple crews in parallel""" + results = await asyncio.gather( + SecurityCrew().crew().akickoff(inputs=self.state.data), + PerformanceCrew().crew().akickoff(inputs=self.state.data), + StyleCrew().crew().akickoff(inputs=self.state.data) + ) + + self.state.security_result = results[0].raw + self.state.performance_result = results[1].raw + self.state.style_result = results[2].raw + return results + + @listen(fan_out_analysis) + def gather_and_synthesize(self, parallel_results): + """Aggregate all parallel results""" + llm = LLM(model="gpt-4o") + self.state.final_report = llm.call([ + {"role": "user", "content": f"""Combine analyses: + Security: {self.state.security_result} + Performance: {self.state.performance_result} + Style: {self.state.style_result} + """} + ]) + return self.state.final_report +``` + +**Batch processing with akickoff_for_each:** +```python +async def batch_analysis(self): + datasets = [{"id": 1, ...}, {"id": 2, ...}, {"id": 3, ...}] + results = await AnalysisCrew().crew().akickoff_for_each(datasets) + return results +``` + +--- + +## Task Guardrails Pattern + +**Use when:** Task outputs must meet specific validation criteria before acceptance. + +**Key feature:** Automatic retry with feedback when validation fails. + +```python +from typing import Tuple, Any +from crewai import Task, TaskOutput + +def validate_json_output(result: TaskOutput) -> Tuple[bool, Any]: + """Validate output is valid JSON""" + try: + data = json.loads(result.raw) + return (True, data) + except json.JSONDecodeError as e: + return (False, f"Invalid JSON: {e}") + +def validate_length(result: TaskOutput) -> Tuple[bool, Any]: + """Validate minimum length""" + if len(result.raw) < 100: + return (False, "Output too short, needs more detail") + return (True, result.raw) + +def validate_no_pii(result: TaskOutput) -> Tuple[bool, Any]: + """Check for PII""" + pii_patterns = ["SSN:", "credit card"] + for pattern in pii_patterns: + if pattern.lower() in result.raw.lower(): + return (False, f"Contains PII pattern: {pattern}") + return (True, result.raw) + +# Single guardrail +task = Task( + description="Generate JSON report", + expected_output="Valid JSON object", + agent=analyst, + guardrail=validate_json_output, + guardrail_max_retries=3 # Retry up to 3x if validation fails +) + +# Multiple sequential guardrails +task = Task( + description="Generate customer report", + expected_output="Detailed report without PII", + agent=writer, + guardrails=[validate_length, validate_no_pii], # Run in order + guardrail_max_retries=3 +) +``` + +**When validation fails:** The agent receives feedback and retries automatically. + +--- + +## State Persistence Pattern + +**Use when:** Long-running flows need crash recovery, or HITL requires resume capability. + +```python +from crewai.flow.flow import Flow, start, listen +from crewai.flow.persistence import persist +from pydantic import BaseModel + +class LongRunningState(BaseModel): + step: str = "initialized" + checkpoint_data: dict = {} + results: list[str] = [] + +@persist() # Saves state after EVERY method +class ResilientFlow(Flow[LongRunningState]): + + @start() + def phase_one(self): + self.state.step = "phase_one_complete" + self.state.checkpoint_data["phase1"] = "data" + # If crash here, flow resumes from this state + return "Phase 1 done" + + @listen(phase_one) + def phase_two(self, prev): + self.state.step = "phase_two_complete" + # State automatically persisted + return "Phase 2 done" + + @listen(phase_two) + def phase_three(self, prev): + self.state.step = "complete" + return "All phases done" + +# First run (crashes at phase_two) +flow1 = ResilientFlow() +result1 = flow1.kickoff() + +# Second run - automatically resumes from persisted state +flow2 = ResilientFlow() +result2 = flow2.kickoff() # Continues from last checkpoint +``` + +**Integration with HITL:** +```python +@persist() +class ApprovalFlow(Flow[ApprovalState]): + + @listen(generate_content) + def await_approval(self, content): + self.state.awaiting_approval = True + self.state.pending_content = content + # State persisted - can resume after human approves + return "Awaiting human approval" +``` + +--- + +## HITL Webhooks Pattern (Enterprise) + +**Use when:** Enterprise deployments require human approval workflows with external system integration. + +```python +import requests + +BASE_URL = "https://your-crewai-deployment.com" + +# Kickoff with human input webhook +response = requests.post(f"{BASE_URL}/kickoff", json={ + "inputs": {"topic": "Quarterly Report"}, + "humanInputWebhook": { + "url": "https://your-app.com/hitl-callback", + "authentication": { + "strategy": "bearer", + "token": "your-secret-token" + } + } +}) + +execution_id = response.json()["execution_id"] + +# When human reviews (via your webhook handler) +# Resume with approval/feedback +def resume_after_human_review(execution_id, task_id, approved, feedback=""): + response = requests.post(f"{BASE_URL}/resume", json={ + "execution_id": execution_id, + "task_id": task_id, + "human_feedback": feedback, + "is_approve": approved + }) + return response.json() + +# Approve and continue +resume_after_human_review(execution_id, "review_task", True, "Looks good!") + +# Reject and retry +resume_after_human_review(execution_id, "review_task", False, "Needs more data on Q3") +``` + +--- + +## Custom Manager Pattern + +**Use when:** Hierarchical coordination requires specialized management behavior. + +**`manager_agent` vs `manager_llm`:** +- Use `manager_llm` for simple coordination with default behavior +- Use `manager_agent` for custom role, goals, and delegation rules + +```python +from crewai import Agent, Crew, Task, Process + +# Custom manager with specific behavior +manager = Agent( + role="Senior Project Manager", + goal="Coordinate team efficiently, prioritize quality over speed", + backstory="""Experienced PM who excels at delegation. + Always validates work before final delivery. + Escalates blockers immediately.""", + allow_delegation=True, # Required for manager + verbose=True +) + +# Specialist agents +researcher = Agent( + role="Research Analyst", + goal="Provide accurate, thorough research", + allow_delegation=False # Specialists don't delegate +) + +writer = Agent( + role="Technical Writer", + goal="Create clear, accurate documentation", + allow_delegation=False +) + +# Hierarchical crew with custom manager +crew = Crew( + agents=[manager, researcher, writer], + tasks=[ + Task(description="Research and document API changes", agent=manager) + ], + process=Process.hierarchical, + manager_agent=manager # Use custom manager +) + +# Alternative: Use LLM as manager (simpler) +crew_simple = Crew( + agents=[researcher, writer], + tasks=[...], + process=Process.hierarchical, + manager_llm="gpt-4o" # Default manager behavior +) +``` + +--- + +## Post-Processing Pattern + +**Use when:** Final output requires dedicated processing (citations, formatting, compliance). + +```python +class PostProcessingFlow(Flow[ContentState]): + + @listen(research_complete) + def add_citations(self, raw_research): + """Dedicated agent for citations""" + citation_agent = Agent( + role="Citation Specialist", + goal="Add proper academic citations", + backstory="Editor with expertise in attribution", + llm=LLM(model="gpt-4o") + ) + + result = citation_agent.kickoff( + f"Add inline citations [Author, Year] to:\n{raw_research.raw}\n" + f"Sources:\n{json.dumps(self.state.sources)}" + ) + self.state.cited_content = result.raw + return result + + @listen(add_citations) + def format_for_publication(self, cited): + """Formatting specialist""" + formatter = Agent( + role="Publication Formatter", + goal="Format for target publication", + llm=LLM(model="gpt-4o") + ) + return formatter.kickoff(f"Format for {self.state.target_format}:\n{cited.raw}") +``` + +--- + +## Composite Patterns + +**Use when:** Enterprise applications require multiple patterns working together. + +### Example: Customer Support System + +Combines: Coordinator, Parallel Fan-Out, Generator-Critic, Task Guardrails, State Persistence, HITL. + +```python +from crewai.flow.flow import Flow, start, listen, router, or_ +from crewai.flow.persistence import persist +from crewai import Agent, Crew, Task, Process +import asyncio + +class SupportState(BaseModel): + ticket: dict = {} + category: str = "" + parallel_analyses: dict = {} + draft_response: str = "" + quality_score: float = 0.0 + iteration: int = 0 + escalated: bool = False + +@persist() # Crash recovery +class CustomerSupportFlow(Flow[SupportState]): + + @start() + def classify_ticket(self): + """Route to appropriate specialist""" + llm = LLM(model="gpt-4o", response_format=TicketClassification) + result = llm.call([ + {"role": "user", "content": f"Classify: {self.state.ticket}"} + ]) + self.state.category = result.category + return result + + @router(classify_ticket) + def route_to_specialist(self): + if self.state.category in ["billing", "technical", "account"]: + return self.state.category + return "general" + + @listen("billing") + async def handle_billing(self): + """Parallel analysis for billing issues""" + results = await asyncio.gather( + BillingCrew().crew().akickoff(inputs=self.state.ticket), + ComplianceCrew().crew().akickoff(inputs=self.state.ticket) + ) + self.state.parallel_analyses = { + "billing": results[0].raw, + "compliance": results[1].raw + } + return results + + @listen("technical") + def handle_technical(self): + """Hierarchical crew for technical issues""" + manager = Agent(role="Tech Lead", allow_delegation=True) + crew = Crew( + agents=[manager, Agent(role="Backend Expert"), Agent(role="Frontend Expert")], + tasks=[Task(description=f"Resolve: {self.state.ticket}", agent=manager)], + process=Process.hierarchical, + manager_agent=manager + ) + result = crew.kickoff() + self.state.parallel_analyses["technical"] = result.raw + return result + + @listen(or_("billing", "technical", "account", "general")) + def generate_response(self): + """Generator creates response""" + crew = ResponseCrew().crew() + result = crew.kickoff(inputs={ + "ticket": self.state.ticket, + "analyses": self.state.parallel_analyses + }) + self.state.draft_response = result.raw + self.state.iteration += 1 + return result + + @listen(generate_response) + def critique_response(self, draft): + """Critic evaluates quality""" + # Task guardrail validates tone + task = Task( + description="Critique response for tone and accuracy", + agent=QACritic(), + guardrail=validate_professional_tone, + guardrail_max_retries=2 + ) + crew = Crew(agents=[QACritic()], tasks=[task]) + result = crew.kickoff(inputs={"response": self.state.draft_response}) + self.state.quality_score = result.pydantic.score + return result + + @router(critique_response) + def quality_gate(self): + if self.state.quality_score >= 0.85: + return "approved" + elif self.state.iteration >= 2: + return "escalate" # HITL escalation + return "revise" + + @listen("revise") + def revise_response(self): + # Loop back to generator with feedback + return self.generate_response() + + @listen("escalate") + def escalate_to_human(self): + """HITL escalation""" + self.state.escalated = True + return {"status": "escalated", "draft": self.state.draft_response} + + @listen("approved") + def send_response(self): + return {"status": "sent", "response": self.state.draft_response} +``` + +--- + +## Anti-Patterns + +### 1. Over-Engineering Simple Tasks +**Wrong:** Using Orchestrator-Worker for a simple classification task. +**Right:** Use direct LLM call with structured output. + +### 2. Missing Exit Conditions +**Wrong:** Iterative loop without max_iterations. +**Right:** Always include `max_iterations` and check in router. + +### 3. Not Using @persist() in Production +**Wrong:** Long-running flow without persistence. +**Right:** Add `@persist()` to any flow that could fail mid-execution. + +### 4. Synchronous When Async Available +**Wrong:** `crew.kickoff()` for multiple independent crews. +**Right:** `await asyncio.gather(*[crew.akickoff() for crew in crews])`. + +### 5. Forgetting Guardrails for Critical Outputs +**Wrong:** Financial report task without validation. +**Right:** Add `guardrail` for compliance and accuracy checks. + +### 6. Skipping Post-Processing +**Wrong:** Sending raw LLM output to customers. +**Right:** Add citation, formatting, and compliance agents. diff --git a/.claude/skills/crewai-enterprise-endpoint-manager/SKILL.md b/.claude/skills/crewai-enterprise-endpoint-manager/SKILL.md new file mode 100644 index 000000000..82d410ce1 --- /dev/null +++ b/.claude/skills/crewai-enterprise-endpoint-manager/SKILL.md @@ -0,0 +1,228 @@ +--- +name: crewai-enterprise-endpoint-manager +description: | + Guide for interacting with deployed CrewAI Enterprise (AOP) crews and flows via API endpoints. + Use this skill when: (1) Kicking off deployed crews/flows programmatically, (2) Monitoring execution status and progress, + (3) Retrieving results from completed executions, (4) Understanding the REST API workflow for deployed agentic workflows, + (5) Implementing authentication with Bearer tokens, (6) Building integrations that consume deployed CrewAI endpoints, + (7) Handling human-in-the-loop webhooks with deployed crews, (8) Managing concurrent executions with semaphore patterns. +--- + +# CrewAI Enterprise Endpoint Manager + +Deployed crews and flows in CrewAI Enterprise (AOP) are accessible via REST API endpoints for programmatic execution, monitoring, and result retrieval. + +## Contents + +- [API Workflow](#api-workflow-overview) +- [Authentication](#authentication) +- [Endpoint Reference](#endpoint-reference) +- [Python Integration](#python-integration) +- [Human-in-the-Loop](#human-in-the-loop-hitl-webhooks) +- [Flow Integration](#integration-with-crewai-flows) +- [Best Practices](#best-practices) +- [Status States](#status-states-reference) + +**Detailed References:** +- [references/python-client.md](references/python-client.md) - Full Python client class, async patterns, batch execution with semaphore +- [references/error-handling.md](references/error-handling.md) - Retry strategies, rate limiting, circuit breaker patterns + +## API Workflow Overview + +``` +1. GET /inputs → Discover required input parameters +2. POST /kickoff → Start execution (returns kickoff_id) +3. GET /{id}/status → Monitor progress and retrieve results +``` + +## Authentication + +All requests require a Bearer token from the **Status tab** of your crew's detail page in the AOP dashboard. + +```bash +curl -H "Authorization: Bearer YOUR_CREW_TOKEN" \ + https://your-crew-url.crewai.com/endpoint +``` + +## Endpoint Reference + +### 1. Discover Inputs - `GET /inputs` + +```bash +curl -H "Authorization: Bearer YOUR_CREW_TOKEN" \ + https://your-crew-url.crewai.com/inputs +``` + +**Response:** +```json +{ + "inputs": [ + {"name": "topic", "type": "string", "required": true}, + {"name": "max_results", "type": "integer", "required": false} + ] +} +``` + +### 2. Start Execution - `POST /kickoff` + +```bash +curl -X POST \ + -H "Authorization: Bearer YOUR_CREW_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"inputs": {"topic": "AI Research", "max_results": 10}}' \ + https://your-crew-url.crewai.com/kickoff +``` + +**Response:** +```json +{"kickoff_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef"} +``` + +### 3. Monitor Status - `GET /{kickoff_id}/status` + +```bash +curl -H "Authorization: Bearer YOUR_CREW_TOKEN" \ + https://your-crew-url.crewai.com/a1b2c3d4-e5f6-7890-1234-567890abcdef/status +``` + +**Running:** +```json +{ + "status": "running", + "current_task": "research_task", + "progress": {"completed_tasks": 1, "total_tasks": 3} +} +``` + +**Completed:** +```json +{ + "status": "completed", + "result": { + "output": "Final output from the crew...", + "tasks": [ + {"task_id": "research_task", "output": "Research findings...", "agent": "Travel Researcher", "execution_time": 45.2} + ] + }, + "execution_time": 108.5 +} +``` + +**Error:** +```json +{"status": "error", "error_message": "Failed to complete task due to invalid input."} +``` + +## Python Integration + +Basic synchronous pattern for simple integrations: + +```python +import requests +import time + +BASE_URL = "https://your-crew-url.crewai.com" +TOKEN = "YOUR_CREW_TOKEN" +headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"} + +def kickoff_crew(inputs: dict) -> str: + """Start execution and return kickoff_id.""" + resp = requests.post(f"{BASE_URL}/kickoff", headers=headers, json={"inputs": inputs}) + resp.raise_for_status() + return resp.json()["kickoff_id"] + +def wait_for_completion(kickoff_id: str, poll_interval: float = 2.0) -> dict: + """Poll until execution completes or fails.""" + while True: + resp = requests.get(f"{BASE_URL}/{kickoff_id}/status", headers=headers) + resp.raise_for_status() + status = resp.json() + if status["status"] in ("completed", "error"): + return status + time.sleep(poll_interval) + +# Usage +kickoff_id = kickoff_crew({"topic": "AI Research"}) +result = wait_for_completion(kickoff_id) +print(result["result"]["output"]) +``` + +**For advanced patterns, see [references/python-client.md](references/python-client.md):** +- Full `CrewAIClient` class with sync/async methods +- Batch execution with semaphore-controlled concurrency +- Rate limiting and progress callbacks +- Structured output parsing with Pydantic + +**For error handling, see [references/error-handling.md](references/error-handling.md):** +- Exponential backoff and retry strategies +- Rate limit handling with adaptive concurrency +- Circuit breaker pattern for resilience + +## Human-in-the-Loop (HITL) Webhooks + +For crews requiring human input during execution: + +```bash +curl -X POST {BASE_URL}/kickoff \ + -H "Authorization: Bearer YOUR_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "inputs": {"topic": "AI Research"}, + "humanInputWebhook": { + "url": "https://your-webhook.com/hitl", + "authentication": {"strategy": "bearer", "token": "your-webhook-secret"} + } + }' +``` + +**Webhook payload received:** +```json +{"kickoff_id": "abc123", "task_id": "review_task", "prompt": "Please review...", "context": {...}} +``` + +**Respond:** +```json +{"response": "Approved with minor edits...", "continue": true} +``` + +## Integration with CrewAI Flows + +Deployed Flows expose the same API. Inputs map to your Flow's state model: + +```python +# Local Flow definition +class ResearchState(BaseModel): + topic: str = "" + depth: int = 1 + +class ResearchFlow(Flow[ResearchState]): + @start() + def begin_research(self): + ... +``` + +```bash +# API call maps to state fields +curl -X POST -H "Authorization: Bearer YOUR_FLOW_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"inputs": {"topic": "Quantum Computing", "depth": 3}}' \ + https://your-flow-url.crewai.com/kickoff +``` + +## Best Practices + +1. **Discover inputs first** - Always call `GET /inputs` to understand required parameters +2. **Handle all status states** - Check for "running", "completed", and "error" +3. **Use semaphores for batches** - Limit concurrent executions (see [python-client.md](references/python-client.md)) +4. **Implement exponential backoff** - For retries: `2^attempt` seconds (see [error-handling.md](references/error-handling.md)) +5. **Store kickoff_ids** - Persist IDs for debugging and resumption +6. **Set appropriate timeouts** - Long-running crews may need 10+ minute timeouts + +## Status States Reference + +| Status | Meaning | Next Action | +|--------|---------|-------------| +| `pending` | Queued for execution | Continue polling | +| `running` | Execution in progress | Continue polling | +| `completed` | Successfully finished | Extract results | +| `error` | Execution failed | Check error_message, retry if transient | diff --git a/.claude/skills/crewai-enterprise-endpoint-manager/references/error-handling.md b/.claude/skills/crewai-enterprise-endpoint-manager/references/error-handling.md new file mode 100644 index 000000000..d00d6d5c8 --- /dev/null +++ b/.claude/skills/crewai-enterprise-endpoint-manager/references/error-handling.md @@ -0,0 +1,450 @@ +# Error Handling Patterns for CrewAI Enterprise API + +Robust error handling strategies for production deployments. + +## Contents + +- [Error Types](#error-types) - HTTP status codes and execution errors +- [Retry Strategies](#retry-strategies) - Exponential backoff, selective retry +- [Rate Limit Handling](#rate-limit-handling) - Detect limits, adaptive concurrency +- [Circuit Breaker Pattern](#circuit-breaker-pattern) - Prevent cascading failures +- [Error Aggregation](#error-aggregation-and-reporting) - Collect and summarize batch errors +- [Timeout Handling](#timeout-handling) - Separate execution and poll timeouts +- [Resilient Batch Execution](#complete-error-resilient-batch-execution) - Full pattern with auto-retry + +## Error Types + +### HTTP Status Errors + +| Status Code | Meaning | Action | +|-------------|---------|--------| +| 400 | Bad Request | Check input format, fix and retry | +| 401 | Unauthorized | Token invalid/expired, refresh token | +| 403 | Forbidden | No access to this crew | +| 404 | Not Found | Crew URL or kickoff_id doesn't exist | +| 429 | Rate Limited | Implement backoff, reduce concurrency | +| 500 | Server Error | Retry with exponential backoff | +| 502/503 | Service Unavailable | Wait and retry | + +### Execution Errors + +```json +{ + "status": "error", + "error_message": "Failed to complete task due to invalid input." +} +``` + +Common causes: +- Invalid inputs for the crew's expected schema +- Agent task failures +- LLM API errors (rate limits, timeouts) +- Tool execution failures + +## Retry Strategies + +### Exponential Backoff + +```python +import asyncio +import random + +async def retry_with_backoff( + func, + max_retries: int = 3, + base_delay: float = 1.0, + max_delay: float = 60.0, + jitter: bool = True +): + """ + Retry with exponential backoff and optional jitter. + + Args: + func: Async function to retry + max_retries: Maximum retry attempts + base_delay: Initial delay in seconds + max_delay: Maximum delay cap + jitter: Add randomness to prevent thundering herd + """ + for attempt in range(max_retries + 1): + try: + return await func() + except Exception as e: + if attempt == max_retries: + raise + + delay = min(base_delay * (2 ** attempt), max_delay) + if jitter: + delay = delay * (0.5 + random.random()) + + print(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay:.1f}s") + await asyncio.sleep(delay) +``` + +### Selective Retry + +```python +from httpx import HTTPStatusError + +RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504} + +async def selective_retry(func, max_retries: int = 3): + """Only retry on specific error types.""" + for attempt in range(max_retries + 1): + try: + return await func() + except HTTPStatusError as e: + if e.response.status_code not in RETRYABLE_STATUS_CODES: + raise # Don't retry 400, 401, 403, 404 + if attempt == max_retries: + raise + await asyncio.sleep(2 ** attempt) + except (asyncio.TimeoutError, ConnectionError) as e: + if attempt == max_retries: + raise + await asyncio.sleep(2 ** attempt) +``` + +## Rate Limit Handling + +### Detect and Respect Rate Limits + +```python +import asyncio +from datetime import datetime, timedelta + +class RateLimitHandler: + """Handle 429 responses with retry-after headers.""" + + def __init__(self): + self.blocked_until: datetime | None = None + + async def wait_if_blocked(self): + """Wait if currently rate limited.""" + if self.blocked_until and datetime.now() < self.blocked_until: + wait_time = (self.blocked_until - datetime.now()).total_seconds() + print(f"Rate limited. Waiting {wait_time:.1f}s") + await asyncio.sleep(wait_time) + + def handle_response(self, response): + """Check response for rate limit headers.""" + if response.status_code == 429: + retry_after = response.headers.get("Retry-After", "60") + try: + seconds = int(retry_after) + except ValueError: + seconds = 60 + + self.blocked_until = datetime.now() + timedelta(seconds=seconds) + raise RateLimitError(f"Rate limited for {seconds}s") + +class RateLimitError(Exception): + pass +``` + +### Adaptive Concurrency + +```python +class AdaptiveSemaphore: + """ + Automatically reduce concurrency on rate limits. + """ + + def __init__(self, initial: int = 10, minimum: int = 1): + self.current = initial + self.minimum = minimum + self.semaphore = asyncio.Semaphore(initial) + self.lock = asyncio.Lock() + self._rate_limit_count = 0 + + async def acquire(self): + await self.semaphore.acquire() + + def release(self): + self.semaphore.release() + + async def reduce_on_rate_limit(self): + """Call when hitting rate limit.""" + async with self.lock: + self._rate_limit_count += 1 + if self._rate_limit_count >= 3 and self.current > self.minimum: + new_value = max(self.current // 2, self.minimum) + print(f"Reducing concurrency: {self.current} -> {new_value}") + self.current = new_value + self.semaphore = asyncio.Semaphore(new_value) + self._rate_limit_count = 0 +``` + +## Circuit Breaker Pattern + +Prevent cascading failures by stopping requests when error rate is too high. + +```python +from dataclasses import dataclass +from datetime import datetime, timedelta +from enum import Enum + +class CircuitState(Enum): + CLOSED = "closed" # Normal operation + OPEN = "open" # Failing, reject requests + HALF_OPEN = "half_open" # Testing if recovered + +@dataclass +class CircuitBreaker: + """ + Circuit breaker to prevent cascading failures. + + Usage: + breaker = CircuitBreaker(failure_threshold=5, recovery_time=30) + + async def make_request(): + if not breaker.allow_request(): + raise CircuitOpenError("Circuit is open") + + try: + result = await client.kickoff(inputs) + breaker.record_success() + return result + except Exception as e: + breaker.record_failure() + raise + """ + failure_threshold: int = 5 + recovery_time: float = 30.0 # seconds + + state: CircuitState = CircuitState.CLOSED + failure_count: int = 0 + last_failure_time: datetime | None = None + + def allow_request(self) -> bool: + """Check if request is allowed.""" + if self.state == CircuitState.CLOSED: + return True + + if self.state == CircuitState.OPEN: + if self._should_attempt_recovery(): + self.state = CircuitState.HALF_OPEN + return True + return False + + # HALF_OPEN: allow one request to test + return True + + def record_success(self): + """Record successful request.""" + self.failure_count = 0 + self.state = CircuitState.CLOSED + + def record_failure(self): + """Record failed request.""" + self.failure_count += 1 + self.last_failure_time = datetime.now() + + if self.state == CircuitState.HALF_OPEN: + self.state = CircuitState.OPEN + elif self.failure_count >= self.failure_threshold: + self.state = CircuitState.OPEN + print(f"Circuit opened after {self.failure_count} failures") + + def _should_attempt_recovery(self) -> bool: + if self.last_failure_time is None: + return True + elapsed = (datetime.now() - self.last_failure_time).total_seconds() + return elapsed >= self.recovery_time + + +class CircuitOpenError(Exception): + pass +``` + +## Error Aggregation and Reporting + +```python +from dataclasses import dataclass, field +from collections import defaultdict + +@dataclass +class ErrorAggregator: + """Collect and summarize errors from batch executions.""" + + errors: list[tuple[dict, str]] = field(default_factory=list) + error_counts: dict[str, int] = field(default_factory=lambda: defaultdict(int)) + + def add_error(self, inputs: dict, error_message: str): + """Record an error.""" + self.errors.append((inputs, error_message)) + + # Categorize error + category = self._categorize(error_message) + self.error_counts[category] += 1 + + def _categorize(self, error_message: str) -> str: + """Categorize error for aggregation.""" + lower = error_message.lower() + if "rate limit" in lower or "429" in lower: + return "rate_limit" + elif "timeout" in lower: + return "timeout" + elif "401" in lower or "unauthorized" in lower: + return "auth" + elif "invalid input" in lower: + return "invalid_input" + elif "500" in lower or "server error" in lower: + return "server_error" + return "other" + + def get_summary(self) -> dict: + """Get error summary.""" + return { + "total_errors": len(self.errors), + "by_category": dict(self.error_counts), + "sample_errors": self.errors[:5] # First 5 for inspection + } + + def get_retryable_inputs(self) -> list[dict]: + """Get inputs that failed with retryable errors.""" + retryable_categories = {"rate_limit", "timeout", "server_error"} + return [ + inputs for inputs, error in self.errors + if self._categorize(error) in retryable_categories + ] + + +# Usage +aggregator = ErrorAggregator() + +for result in batch_results: + if result.error: + aggregator.add_error(result.inputs, result.error) + +summary = aggregator.get_summary() +print(f"Total errors: {summary['total_errors']}") +print(f"By category: {summary['by_category']}") + +# Retry failed inputs +retryable = aggregator.get_retryable_inputs() +if retryable: + print(f"Retrying {len(retryable)} failed executions...") + retry_results = await client.kickoff_batch(retryable, max_concurrent=5) +``` + +## Timeout Handling + +```python +async def kickoff_with_timeout( + client, + inputs: dict, + execution_timeout: float = 600.0, + poll_timeout: float = 10.0 +) -> ExecutionResult: + """ + Kickoff with separate timeouts for execution and polling. + + Args: + execution_timeout: Max total time for the crew to complete + poll_timeout: Timeout for each status poll request + """ + start_time = datetime.now() + + async with httpx.AsyncClient(timeout=30.0) as http_client: + # Kickoff (short timeout) + resp = await http_client.post( + f"{client.base_url}/kickoff", + headers=client._headers, + json={"inputs": inputs} + ) + kickoff_id = resp.json()["kickoff_id"] + + # Poll with timeout + while True: + elapsed = (datetime.now() - start_time).total_seconds() + if elapsed > execution_timeout: + return ExecutionResult( + kickoff_id=kickoff_id, + inputs=inputs, + status=ExecutionStatus.ERROR, + error=f"Execution timeout after {elapsed:.0f}s" + ) + + try: + async with asyncio.timeout(poll_timeout): + resp = await http_client.get( + f"{client.base_url}/{kickoff_id}/status", + headers=client._headers + ) + except asyncio.TimeoutError: + continue # Retry poll + + data = resp.json() + if data["status"] in ("completed", "error"): + return ExecutionResult(...) + + await asyncio.sleep(2) +``` + +## Complete Error-Resilient Batch Execution + +```python +async def resilient_batch_execution( + client: CrewAIClient, + inputs_list: list[dict], + max_concurrent: int = 10, + max_total_retries: int = 3 +) -> tuple[list[ExecutionResult], list[ExecutionResult]]: + """ + Execute batch with automatic retry of failed executions. + + Returns: + Tuple of (successful_results, final_failed_results) + """ + all_successful = [] + remaining = inputs_list.copy() + + for retry_round in range(max_total_retries + 1): + if not remaining: + break + + if retry_round > 0: + print(f"Retry round {retry_round}: {len(remaining)} executions") + # Reduce concurrency on retries + current_concurrent = max(max_concurrent // (2 ** retry_round), 1) + else: + current_concurrent = max_concurrent + + results = await client.kickoff_batch( + remaining, + max_concurrent=current_concurrent + ) + + # Separate successful and failed + successful = [r for r in results if r.is_success] + failed = [r for r in results if not r.is_success] + + all_successful.extend(successful) + + # Get retryable failures + aggregator = ErrorAggregator() + for r in failed: + aggregator.add_error(r.inputs, r.error or "Unknown error") + + remaining = aggregator.get_retryable_inputs() + + if not remaining: + # All remaining failures are non-retryable + return all_successful, failed + + await asyncio.sleep(5 * (retry_round + 1)) # Increasing wait between rounds + + # Return final state + final_failed = [ + ExecutionResult( + kickoff_id="", + inputs=inputs, + status=ExecutionStatus.ERROR, + error="Max retries exceeded" + ) + for inputs in remaining + ] + + return all_successful, final_failed +``` diff --git a/.claude/skills/crewai-enterprise-endpoint-manager/references/python-client.md b/.claude/skills/crewai-enterprise-endpoint-manager/references/python-client.md new file mode 100644 index 000000000..5f965d0b9 --- /dev/null +++ b/.claude/skills/crewai-enterprise-endpoint-manager/references/python-client.md @@ -0,0 +1,648 @@ +# CrewAI Enterprise Python Client + +Complete Python client implementation for interacting with deployed CrewAI crews and flows. + +## Contents + +- [Full-Featured Client Class](#full-featured-client-class) - Production-ready `CrewAIClient` with sync/async methods +- [Usage Examples](#usage-examples) - Basic, async, structured output, error handling +- [Environment Configuration](#environment-configuration) - Config from environment variables +- [Rate-Limited Semaphore](#rate-limited-semaphore) - Combine concurrency with rate limiting +- [Quick Start Examples](#quick-start-examples) - Minimal code to get started +- [Dependencies](#dependencies) - Required packages + +## Full-Featured Client Class + +```python +""" +CrewAI Enterprise API Client + +Production-ready client for interacting with deployed crews and flows. +Supports synchronous, async, and batch operations with semaphore control. +""" + +import asyncio +import httpx +import logging +from dataclasses import dataclass, field +from datetime import datetime +from typing import Any, Callable, TypeVar, Generic +from enum import Enum +from pydantic import BaseModel + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +T = TypeVar("T", bound=BaseModel) + + +class ExecutionStatus(str, Enum): + PENDING = "pending" + RUNNING = "running" + COMPLETED = "completed" + ERROR = "error" + + +@dataclass +class ExecutionResult: + """Result of a single crew execution.""" + kickoff_id: str + inputs: dict + status: ExecutionStatus + result: dict | None = None + error: str | None = None + started_at: datetime | None = None + completed_at: datetime | None = None + progress: dict | None = None + + @property + def duration_seconds(self) -> float | None: + if self.started_at and self.completed_at: + return (self.completed_at - self.started_at).total_seconds() + return None + + @property + def is_success(self) -> bool: + return self.status == ExecutionStatus.COMPLETED and self.result is not None + + +@dataclass +class CrewAIClient: + """ + Client for CrewAI Enterprise API. + + Usage: + client = CrewAIClient( + base_url="https://your-crew.crewai.com", + token="YOUR_TOKEN" + ) + + # Sync usage + result = client.kickoff_sync({"topic": "AI"}) + + # Async usage + result = await client.kickoff({"topic": "AI"}) + + # Batch with semaphore + results = await client.kickoff_batch(inputs_list, max_concurrent=10) + """ + base_url: str + token: str + timeout: float = 600.0 + poll_interval: float = 2.0 + max_retries: int = 3 + + _headers: dict = field(init=False) + + def __post_init__(self): + self.base_url = self.base_url.rstrip("/") + self._headers = { + "Authorization": f"Bearer {self.token}", + "Content-Type": "application/json" + } + + # ========================================================================= + # Input Discovery + # ========================================================================= + + async def get_inputs(self) -> dict: + """Discover required inputs for the crew.""" + async with httpx.AsyncClient(timeout=30.0) as client: + resp = await client.get( + f"{self.base_url}/inputs", + headers=self._headers + ) + resp.raise_for_status() + return resp.json() + + def get_inputs_sync(self) -> dict: + """Synchronous version of get_inputs.""" + import requests + resp = requests.get( + f"{self.base_url}/inputs", + headers=self._headers, + timeout=30 + ) + resp.raise_for_status() + return resp.json() + + # ========================================================================= + # Single Execution + # ========================================================================= + + async def kickoff( + self, + inputs: dict, + wait: bool = True, + webhook_config: dict | None = None + ) -> ExecutionResult: + """ + Start a crew execution. + + Args: + inputs: Input parameters for the crew + wait: If True, poll until completion + webhook_config: Optional HITL webhook configuration + + Returns: + ExecutionResult with status and results + """ + started_at = datetime.now() + + async with httpx.AsyncClient(timeout=self.timeout) as client: + # Build request body + body: dict[str, Any] = {"inputs": inputs} + if webhook_config: + body["humanInputWebhook"] = webhook_config + + # Kickoff + resp = await client.post( + f"{self.base_url}/kickoff", + headers=self._headers, + json=body + ) + resp.raise_for_status() + kickoff_id = resp.json()["kickoff_id"] + + if not wait: + return ExecutionResult( + kickoff_id=kickoff_id, + inputs=inputs, + status=ExecutionStatus.RUNNING, + started_at=started_at + ) + + # Poll for completion + return await self._poll_until_complete( + client, kickoff_id, inputs, started_at + ) + + def kickoff_sync( + self, + inputs: dict, + wait: bool = True, + webhook_config: dict | None = None + ) -> ExecutionResult: + """Synchronous version of kickoff.""" + return asyncio.run(self.kickoff(inputs, wait, webhook_config)) + + async def get_status(self, kickoff_id: str) -> ExecutionResult: + """Get the current status of an execution.""" + async with httpx.AsyncClient(timeout=30.0) as client: + resp = await client.get( + f"{self.base_url}/{kickoff_id}/status", + headers=self._headers + ) + resp.raise_for_status() + data = resp.json() + + return ExecutionResult( + kickoff_id=kickoff_id, + inputs={}, + status=ExecutionStatus(data["status"]), + result=data.get("result"), + error=data.get("error_message"), + progress=data.get("progress") + ) + + # ========================================================================= + # Batch Execution with Semaphore + # ========================================================================= + + async def kickoff_batch( + self, + inputs_list: list[dict], + max_concurrent: int = 10, + on_progress: Callable[[int, int, ExecutionResult], None] | None = None, + on_error: Callable[[ExecutionResult], bool] | None = None + ) -> list[ExecutionResult]: + """ + Execute multiple crews with semaphore-controlled concurrency. + + Args: + inputs_list: List of input dictionaries + max_concurrent: Maximum concurrent executions + on_progress: Callback(completed, total, result) for progress + on_error: Callback(result) -> bool, return True to continue + + Returns: + List of ExecutionResult objects + """ + semaphore = asyncio.Semaphore(max_concurrent) + results: list[ExecutionResult] = [] + completed_count = 0 + total = len(inputs_list) + lock = asyncio.Lock() + + async def execute_with_semaphore(inputs: dict) -> ExecutionResult: + nonlocal completed_count + + async with semaphore: + result = await self._execute_single_with_retry(inputs) + + async with lock: + completed_count += 1 + results.append(result) + + if on_progress: + on_progress(completed_count, total, result) + + if result.error and on_error: + should_continue = on_error(result) + if not should_continue: + raise asyncio.CancelledError("Stopped by on_error callback") + + logger.info( + f"[{completed_count}/{total}] " + f"{'OK' if result.is_success else 'ERR'} " + f"id={result.kickoff_id}" + ) + + return result + + tasks = [execute_with_semaphore(inputs) for inputs in inputs_list] + + try: + await asyncio.gather(*tasks) + except asyncio.CancelledError: + logger.warning("Batch execution cancelled") + + return results + + def kickoff_batch_sync( + self, + inputs_list: list[dict], + max_concurrent: int = 10, + on_progress: Callable[[int, int, ExecutionResult], None] | None = None + ) -> list[ExecutionResult]: + """Synchronous version of kickoff_batch.""" + return asyncio.run( + self.kickoff_batch(inputs_list, max_concurrent, on_progress) + ) + + # ========================================================================= + # Structured Output Support + # ========================================================================= + + async def kickoff_typed( + self, + inputs: dict, + output_model: type[T] + ) -> T | None: + """ + Execute and parse result into a Pydantic model. + + Args: + inputs: Input parameters + output_model: Pydantic model class for parsing + + Returns: + Parsed model instance or None if failed + """ + result = await self.kickoff(inputs, wait=True) + + if result.is_success and result.result: + output = result.result.get("output", "") + # Try to parse JSON from output + import json + try: + data = json.loads(output) + return output_model.model_validate(data) + except (json.JSONDecodeError, Exception): + return None + return None + + # ========================================================================= + # Private Methods + # ========================================================================= + + async def _execute_single_with_retry(self, inputs: dict) -> ExecutionResult: + """Execute single crew with retry logic.""" + started_at = datetime.now() + + for attempt in range(self.max_retries): + try: + async with httpx.AsyncClient(timeout=self.timeout) as client: + # Kickoff + resp = await client.post( + f"{self.base_url}/kickoff", + headers=self._headers, + json={"inputs": inputs} + ) + resp.raise_for_status() + kickoff_id = resp.json()["kickoff_id"] + + # Poll + return await self._poll_until_complete( + client, kickoff_id, inputs, started_at + ) + + except httpx.HTTPStatusError as e: + if attempt == self.max_retries - 1: + return ExecutionResult( + kickoff_id="", + inputs=inputs, + status=ExecutionStatus.ERROR, + error=f"HTTP {e.response.status_code}: {str(e)}", + started_at=started_at, + completed_at=datetime.now() + ) + await asyncio.sleep(2 ** attempt) + + except Exception as e: + if attempt == self.max_retries - 1: + return ExecutionResult( + kickoff_id="", + inputs=inputs, + status=ExecutionStatus.ERROR, + error=str(e), + started_at=started_at, + completed_at=datetime.now() + ) + await asyncio.sleep(2 ** attempt) + + # Should never reach here + return ExecutionResult( + kickoff_id="", + inputs=inputs, + status=ExecutionStatus.ERROR, + error="Max retries exceeded", + started_at=started_at, + completed_at=datetime.now() + ) + + async def _poll_until_complete( + self, + client: httpx.AsyncClient, + kickoff_id: str, + inputs: dict, + started_at: datetime + ) -> ExecutionResult: + """Poll status until completion or error.""" + while True: + resp = await client.get( + f"{self.base_url}/{kickoff_id}/status", + headers=self._headers + ) + resp.raise_for_status() + data = resp.json() + + status = ExecutionStatus(data["status"]) + + if status == ExecutionStatus.COMPLETED: + return ExecutionResult( + kickoff_id=kickoff_id, + inputs=inputs, + status=status, + result=data.get("result"), + started_at=started_at, + completed_at=datetime.now() + ) + + if status == ExecutionStatus.ERROR: + return ExecutionResult( + kickoff_id=kickoff_id, + inputs=inputs, + status=status, + error=data.get("error_message", "Unknown error"), + started_at=started_at, + completed_at=datetime.now() + ) + + await asyncio.sleep(self.poll_interval) +``` + +## Usage Examples + +### Basic Usage + +```python +# Initialize client +client = CrewAIClient( + base_url="https://your-crew.crewai.com", + token="YOUR_TOKEN" +) + +# Discover inputs +inputs_schema = client.get_inputs_sync() +print(f"Required inputs: {inputs_schema}") + +# Single execution (sync) +result = client.kickoff_sync({"topic": "AI Research"}) +if result.is_success: + print(f"Output: {result.result['output']}") +else: + print(f"Error: {result.error}") +``` + +### Async with Progress Tracking + +```python +import asyncio + +async def main(): + client = CrewAIClient( + base_url="https://your-crew.crewai.com", + token="YOUR_TOKEN", + max_concurrent=10 + ) + + # 100 executions + inputs_list = [{"topic": f"Topic {i}"} for i in range(100)] + + def on_progress(completed, total, result): + pct = completed / total * 100 + status = "OK" if result.is_success else "ERR" + print(f"[{completed}/{total}] {pct:.0f}% - {status}") + + results = await client.kickoff_batch( + inputs_list, + max_concurrent=10, + on_progress=on_progress + ) + + # Summary + success = sum(1 for r in results if r.is_success) + print(f"\nCompleted: {success}/{len(results)}") + +asyncio.run(main()) +``` + +### With Structured Output + +```python +from pydantic import BaseModel + +class ResearchOutput(BaseModel): + summary: str + key_findings: list[str] + confidence: float + +async def main(): + client = CrewAIClient(...) + + output = await client.kickoff_typed( + inputs={"topic": "Quantum Computing"}, + output_model=ResearchOutput + ) + + if output: + print(f"Summary: {output.summary}") + print(f"Findings: {output.key_findings}") +``` + +### Error Handling with Early Stop + +```python +async def main(): + client = CrewAIClient(...) + + error_count = 0 + max_errors = 5 + + def on_error(result): + nonlocal error_count + error_count += 1 + print(f"Error {error_count}: {result.error}") + # Stop if too many errors + return error_count < max_errors + + results = await client.kickoff_batch( + inputs_list, + max_concurrent=10, + on_error=on_error + ) +``` + +## Environment Configuration + +```python +import os +from dataclasses import dataclass + +@dataclass +class Config: + base_url: str = os.getenv("CREWAI_BASE_URL", "") + token: str = os.getenv("CREWAI_TOKEN", "") + max_concurrent: int = int(os.getenv("CREWAI_MAX_CONCURRENT", "10")) + timeout: float = float(os.getenv("CREWAI_TIMEOUT", "600")) + +config = Config() +client = CrewAIClient( + base_url=config.base_url, + token=config.token, + timeout=config.timeout +) +``` + +## Rate-Limited Semaphore + +For APIs with rate limits, combine concurrency control with rate limiting: + +```python +import asyncio +from collections import deque +from time import time + +class RateLimitedSemaphore: + """ + Semaphore with rate limiting: max N concurrent requests AND max M requests per second. + + Usage: + limiter = RateLimitedSemaphore(max_concurrent=10, max_per_second=5.0) + + async def make_request(): + async with limiter: + return await client.kickoff(inputs) + """ + + def __init__(self, max_concurrent: int, max_per_second: float): + self.semaphore = asyncio.Semaphore(max_concurrent) + self.max_per_second = max_per_second + self.request_times: deque = deque() + self.lock = asyncio.Lock() + + async def __aenter__(self): + await self.semaphore.acquire() + + async with self.lock: + now = time() + # Remove timestamps older than 1 second + while self.request_times and now - self.request_times[0] > 1.0: + self.request_times.popleft() + + # If at rate limit, wait + if len(self.request_times) >= self.max_per_second: + sleep_time = 1.0 - (now - self.request_times[0]) + if sleep_time > 0: + await asyncio.sleep(sleep_time) + + self.request_times.append(time()) + + async def __aexit__(self, *args): + self.semaphore.release() + + +# Usage with CrewAIClient +async def batch_with_rate_limit(client: CrewAIClient, inputs_list: list[dict]): + """Execute batch with both concurrency and rate limiting.""" + limiter = RateLimitedSemaphore(max_concurrent=10, max_per_second=5.0) + + async def single_execution(inputs: dict) -> ExecutionResult: + async with limiter: + return await client.kickoff(inputs, wait=True) + + tasks = [single_execution(inputs) for inputs in inputs_list] + return await asyncio.gather(*tasks) +``` + +## Quick Start Examples + +### Minimal Example + +```python +# Simplest possible usage +client = CrewAIClient( + base_url="https://your-crew.crewai.com", + token="YOUR_TOKEN" +) + +result = client.kickoff_sync({"topic": "AI Research"}) +print(result.result["output"] if result.is_success else result.error) +``` + +### Batch Processing 100 Items + +```python +import asyncio + +async def process_batch(): + client = CrewAIClient( + base_url="https://your-crew.crewai.com", + token="YOUR_TOKEN" + ) + + inputs = [{"topic": f"Topic {i}"} for i in range(100)] + + results = await client.kickoff_batch( + inputs, + max_concurrent=10, + on_progress=lambda done, total, r: print(f"{done}/{total}") + ) + + success = sum(1 for r in results if r.is_success) + print(f"Success: {success}/{len(results)}") + +asyncio.run(process_batch()) +``` + +## Dependencies + +```toml +# pyproject.toml +[project] +dependencies = [ + "httpx>=0.25.0", + "pydantic>=2.0.0", +] +``` diff --git a/.claude/skills/crewai-tool-creator/SKILL.md b/.claude/skills/crewai-tool-creator/SKILL.md new file mode 100644 index 000000000..3625e8d8b --- /dev/null +++ b/.claude/skills/crewai-tool-creator/SKILL.md @@ -0,0 +1,448 @@ +--- +name: crewai-tool-creator +description: | + Guide for creating custom tools for CrewAI agents following best practices from both CrewAI documentation + and Anthropic's agent tool design principles. Use this skill when: (1) Creating new custom tools for agents, + (2) Designing tool input schemas with Pydantic, (3) Writing effective tool descriptions and error messages, + (4) Implementing caching and async tools, (5) Optimizing tools for context efficiency and token consumption. + Always use BaseTool class inheritance for full control over validation, error handling, and behavior. +--- + +# CrewAI Tool Creator Guide + +**Core Principle: Design tools that reduce agent cognitive load while enabling clear, distinct actions.** + +Tools are how agents interact with the world. Well-designed tools make agents more effective; poorly designed tools cause confusion, hallucinations, and wasted tokens. + +## BaseTool Class Structure + +Always use the `BaseTool` class for full control over input validation, error handling, and state: + +```python +from typing import Type +from crewai.tools import BaseTool +from pydantic import BaseModel, Field + +class SearchContactsInput(BaseModel): + """Input schema for contact search.""" + query: str = Field(..., description="Name or email to search for (partial match)") + limit: int = Field(default=10, ge=1, le=100, description="Max results (1-100)") + +class SearchContactsTool(BaseTool): + name: str = "search_contacts" + description: str = """ + Search for contacts by name or email. Returns matching contacts with + their name, email, and role. Use this instead of listing all contacts. + """ + args_schema: Type[BaseModel] = SearchContactsInput + + def _run(self, query: str, limit: int = 10) -> str: + try: + results = self.db.search_contacts(query, limit) + return json.dumps(results) + except DatabaseError as e: + return f"Search failed: {e}. Try a different query or check connection." +``` + +## Critical Design Principles + +### 1. Context Efficiency + +Agents have limited context. Design tools that consolidate operations: + +```python +# BAD: Forces multiple calls, wastes context +class ListContactsTool(BaseTool): + name: str = "list_contacts" + description: str = "List all contacts in the database." + + def _run(self) -> str: + return json.dumps(db.get_all()) # Could return thousands + +# GOOD: Search reduces context consumption +class SearchContactsTool(BaseTool): + name: str = "search_contacts" + description: str = "Search contacts by name/email. Max 25 results per query." + args_schema: Type[BaseModel] = SearchInput + + def _run(self, query: str, limit: int = 25) -> str: + return json.dumps(db.search(query, limit)) +``` + +### 2. Clear Purpose and Naming + +Each tool should have one clear purpose with an unambiguous name: + +```python +# BAD: Vague, what does "process" mean? +name: str = "data_tool" + +# GOOD: Clear action and target +name: str = "search_customer_orders" + +# Use namespacing for related tools +name: str = "crm_search_contacts" +name: str = "crm_create_contact" +name: str = "crm_update_contact" +``` + +### 3. Semantic Parameter Names + +Use descriptive names, not cryptic identifiers: + +```python +# BAD: Ambiguous +class BadInput(BaseModel): + user: str # User what? ID? Name? Email? + id: str # ID of what? + +# GOOD: Unambiguous +class GoodInput(BaseModel): + user_email: str = Field(..., description="Email address of the user") + order_id: str = Field(..., description="Order ID (format: ORD-XXXXX)") +``` + +### 4. Meaningful Response Design + +Return what agents need, exclude what they don't: + +```python +# BAD: Dumps everything including useless metadata +def _run(self, order_id: str) -> str: + order = db.get_order(order_id) + return json.dumps(order.__dict__) # Includes uuid, created_at_unix, internal_flags... + +# GOOD: Curated, relevant fields +def _run(self, order_id: str) -> str: + order = db.get_order(order_id) + return json.dumps({ + "order_id": order.display_id, + "customer": order.customer_name, + "items": [{"name": i.name, "qty": i.qty} for i in order.items], + "status": order.status, + "total": f"${order.total:.2f}" + }) +``` + +### 5. Actionable Error Messages + +Replace stack traces with guidance: + +```python +# BAD: Unhelpful +def _run(self, date: str) -> str: + try: + parsed = datetime.fromisoformat(date) + except ValueError: + return "Error: Invalid date format" + +# GOOD: Guides correction +def _run(self, date: str) -> str: + try: + parsed = datetime.fromisoformat(date) + except ValueError: + return ( + f"Invalid date format: '{date}'. " + f"Use ISO 8601 format: YYYY-MM-DD (e.g., 2024-03-15)" + ) +``` + +## Tool Description Best Practices + +Write descriptions as you would for a new team member: + +```python +description: str = """ +Search for orders by customer email or order ID. + +When to use: +- Finding a specific customer's order history +- Looking up order status by ID +- Checking recent orders for a customer + +Returns: Order details including status, items, and total. +Does NOT return: Payment details or internal notes. + +Tip: For bulk order analysis, use 'export_orders' instead. +""" +``` + +## Pydantic Input Schema Patterns + +### Required vs Optional Fields + +```python +class AnalyzeInput(BaseModel): + # Required: no default value + data_source: str = Field(..., description="Data source identifier") + + # Optional with default + include_historical: bool = Field( + default=False, + description="Include historical data (slower)" + ) + + # Optional, can be None + date_filter: Optional[str] = Field( + default=None, + description="Filter by date (ISO format) or None for all" + ) +``` + +### Validation Constraints + +```python +class PaginatedInput(BaseModel): + query: str = Field(..., min_length=2, max_length=200) + page: int = Field(default=1, ge=1, description="Page number (starts at 1)") + per_page: int = Field(default=25, ge=1, le=100, description="Results per page") + + # Complex validation + @field_validator('query') + @classmethod + def validate_query(cls, v: str) -> str: + if v.strip() != v: + raise ValueError("Query cannot have leading/trailing whitespace") + return v +``` + +### Enum for Controlled Values + +```python +from enum import Enum + +class SortOrder(str, Enum): + ASC = "asc" + DESC = "desc" + +class SortableInput(BaseModel): + query: str = Field(...) + sort_by: str = Field(default="created_at", description="Field to sort by") + sort_order: SortOrder = Field(default=SortOrder.DESC) +``` + +## Response Format Flexibility + +Let agents request the detail level they need: + +```python +class AnalyzeDataInput(BaseModel): + data_source: str = Field(..., description="Data source identifier") + response_format: str = Field( + default="concise", + description="'concise' for summary only, 'detailed' for full breakdown" + ) + +class AnalyzeDataTool(BaseTool): + name: str = "analyze_data" + description: str = "Analyze data source. Use response_format='concise' unless full details needed." + args_schema: Type[BaseModel] = AnalyzeDataInput + + def _run(self, data_source: str, response_format: str = "concise") -> str: + analysis = self.analyze(data_source) + + if response_format == "concise": + return json.dumps({ + "summary": analysis.summary, + "key_metrics": analysis.top_3_metrics + }) # ~70 tokens + else: + return json.dumps({ + "summary": analysis.summary, + "all_metrics": analysis.all_metrics, + "data_points": analysis.raw_data, + "methodology": analysis.methodology + }) # ~300 tokens +``` + +## Tool with Constructor Dependencies + +Inject dependencies through `__init__`: + +```python +class DatabaseSearchTool(BaseTool): + name: str = "db_search" + description: str = "Search the database" + args_schema: Type[BaseModel] = SearchInput + + def __init__(self, db_connection, cache_client=None): + super().__init__() + self.db = db_connection + self.cache = cache_client + + def _run(self, query: str) -> str: + if self.cache: + cached = self.cache.get(query) + if cached: + return cached + + results = self.db.search(query) + if self.cache: + self.cache.set(query, results) + return results +``` + +## Tool Caching + +**When to use:** Expensive API calls, idempotent operations, repeated queries with same results. + +Use the `cache_function` attribute to control caching. It receives `(arguments: dict, result)` and returns `bool`. + +```python +from typing import Callable, Any + +class ExpensiveSearchTool(BaseTool): + name: str = "expensive_search" + description: str = "Search with caching for repeated queries" + args_schema: Type[BaseModel] = SearchInput + + # Cache all successful results + cache_function: Callable[[dict, Any], bool] = lambda args, result: ( + not str(result).startswith("Error:") # Don't cache errors + ) + + def _run(self, query: str) -> str: + return expensive_api.search(query) +``` + +**When NOT to cache:** +- User-specific data that changes frequently +- Time-sensitive information +- Error responses (agent should retry) + +See [references/tool-patterns.md](references/tool-patterns.md#tool-caching) for conditional caching patterns. + +## Async Execution + +**Important:** Tool `_run()` methods are **synchronous**. Async execution happens at the crew/task level, not the tool level. + +**When you need parallel operations within a tool**, use `ThreadPoolExecutor`: + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +class MultiSourceTool(BaseTool): + def _run(self, query: str) -> str: + with ThreadPoolExecutor(max_workers=3) as executor: + futures = {executor.submit(src.search, query): src for src in self.sources} + results = {futures[f]: f.result() for f in as_completed(futures)} + return json.dumps(results) +``` + +**For async crew execution**, use `akickoff()` or `kickoff_async()` at the flow level. + +See [references/tool-patterns.md](references/tool-patterns.md#async-execution-patterns) for async patterns. + +## Complete Production Example + +```python +from typing import Type, Optional +from crewai.tools import BaseTool +from pydantic import BaseModel, Field +import json + +class CustomerSearchInput(BaseModel): + """Input schema for customer search tool.""" + query: str = Field( + ..., + min_length=2, + description="Search by name, email, or phone (min 2 chars)" + ) + status: Optional[str] = Field( + default=None, + description="Filter by status: 'active', 'inactive', or None for all" + ) + limit: int = Field( + default=10, + ge=1, + le=50, + description="Results per page (1-50, default 10)" + ) + +class CustomerSearchTool(BaseTool): + name: str = "crm_search_customers" + description: str = """ + Search for customers by name, email, or phone number. + + Use cases: + - Find a specific customer's profile + - Look up customers by partial name/email + - Filter active vs inactive customers + + Returns: Customer name, email, status, and last order date. + For full customer details, use 'crm_get_customer' with the customer_id. + """ + args_schema: Type[BaseModel] = CustomerSearchInput + + def __init__(self, db_connection): + super().__init__() + self.db = db_connection + + def _run(self, query: str, status: Optional[str] = None, limit: int = 10) -> str: + # Validate status if provided + valid_statuses = {"active", "inactive", None} + if status and status not in valid_statuses: + return ( + f"Invalid status: '{status}'. " + f"Use 'active', 'inactive', or omit for all customers." + ) + + try: + results = self.db.search_customers( + query=query, + status=status, + limit=limit + ) + + if not results: + return f"No customers found matching '{query}'" + ( + f" with status '{status}'" if status else "" + ) + ". Try a broader search term." + + # Return curated, agent-friendly format + return json.dumps({ + "count": len(results), + "customers": [ + { + "customer_id": c.id, + "name": c.full_name, + "email": c.email, + "status": c.status, + "last_order": c.last_order_date.isoformat() if c.last_order_date else None + } + for c in results + ] + }) + + except DatabaseError: + return "Search failed: Database unavailable. Retry in a moment or contact support." +``` + +## Assigning Tools to Agents + +```python +from crewai import Agent + +# Initialize tools with dependencies +search_tool = CustomerSearchTool(db_connection=db) +order_tool = OrderLookupTool(db_connection=db) + +# Assign to agent +support_agent = Agent( + role="Customer Support Specialist", + goal="Quickly resolve customer inquiries", + backstory="Expert at navigating customer systems", + tools=[search_tool, order_tool], + verbose=True +) +``` + +## Reference Files + +- [references/tool-patterns.md](references/tool-patterns.md) - Advanced patterns: MCP integration, tool composition, testing +- [references/design-principles.md](references/design-principles.md) - Anthropic's complete agent tool design principles + +## Sources + +- [CrewAI Custom Tools Documentation](https://docs.crewai.com/en/learn/create-custom-tools) +- [Anthropic: Writing Tools for Agents](https://www.anthropic.com/engineering/writing-tools-for-agents) diff --git a/.claude/skills/crewai-tool-creator/references/design-principles.md b/.claude/skills/crewai-tool-creator/references/design-principles.md new file mode 100644 index 000000000..0556cc2f4 --- /dev/null +++ b/.claude/skills/crewai-tool-creator/references/design-principles.md @@ -0,0 +1,404 @@ +# Agent Tool Design Principles + +Based on [Anthropic's Engineering Guide: Writing Tools for Agents](https://www.anthropic.com/engineering/writing-tools-for-agents) + +## Core Philosophy + +> "LLM agents have limited 'context'...whereas computer memory is cheap and abundant" + +Agents operate under fundamentally different constraints than traditional software. Every token consumed reduces their ability to reason. Design tools that maximize value per token. + +## Principle 1: Context Efficiency + +### The Problem +Agents have limited context windows. Every tool response consumes tokens that could be used for reasoning. + +### The Solution +Build tools that consolidate operations and return only what's needed. + +```python +# ANTI-PATTERN: List everything +class ListAllContactsTool(BaseTool): + name: str = "list_contacts" + + def _run(self) -> str: + # Returns 10,000 contacts = 500,000 tokens wasted + return json.dumps(db.get_all_contacts()) + +# PATTERN: Search with limits +class SearchContactsTool(BaseTool): + name: str = "search_contacts" + description: str = "Search contacts. Use specific queries to find who you need." + + def _run(self, query: str, limit: int = 25) -> str: + # Returns max 25 results = ~2,000 tokens + return json.dumps(db.search(query, limit=limit)) +``` + +### Multi-Step Consolidation + +Combine related operations that are typically used together: + +```python +# ANTI-PATTERN: Requires 3 tool calls +# 1. get_user(id) -> user +# 2. get_user_orders(user_id) -> orders +# 3. get_order_details(order_id) -> details + +# PATTERN: Single call with context +class UserContextTool(BaseTool): + name: str = "get_user_context" + description: str = """ + Get user profile with recent activity. + Returns user info, last 5 orders, and active support tickets. + """ + + def _run(self, user_id: str) -> str: + user = db.get_user(user_id) + orders = db.get_recent_orders(user_id, limit=5) + tickets = db.get_open_tickets(user_id) + + return json.dumps({ + "user": {"name": user.name, "email": user.email, "tier": user.tier}, + "recent_orders": [{"id": o.id, "date": o.date, "status": o.status} for o in orders], + "open_tickets": len(tickets) + }) +``` + +## Principle 2: Clear, Distinct Purpose + +### Tool Selection +Agents must choose between available tools. Overlapping purposes cause confusion. + +```python +# ANTI-PATTERN: Overlapping tools +tools = [ + DataTool(), # "Process data" + AnalyzerTool(), # "Analyze data" + ProcessorTool(), # "Handle data processing" +] +# Agent: "Which one do I use?" + +# PATTERN: Distinct purposes +tools = [ + DataValidationTool(), # "Validate data format and completeness" + DataTransformTool(), # "Convert data between formats (CSV, JSON, XML)" + DataAnalysisTool(), # "Calculate statistics and identify patterns" +] +``` + +### Namespacing + +Group related tools with consistent prefixes: + +```python +# Service-based namespacing +name: str = "asana_search_tasks" +name: str = "asana_create_task" +name: str = "asana_update_task" + +# Or resource-based +name: str = "search_asana_tasks" +name: str = "create_asana_task" +name: str = "update_asana_task" +``` + +## Principle 3: Semantic Clarity + +### Parameter Naming + +Agents handle natural language better than cryptic identifiers: + +```python +# ANTI-PATTERN: Ambiguous parameters +class BadInput(BaseModel): + user: str # ID? Email? Name? + id: str # Of what? + type: str # What types exist? + +# PATTERN: Self-documenting parameters +class GoodInput(BaseModel): + user_email: str = Field(..., description="User's email address") + project_id: str = Field(..., description="Project ID (format: PRJ-XXXXX)") + task_type: str = Field( + ..., + description="Type of task: 'bug', 'feature', or 'improvement'" + ) +``` + +### ID Resolution + +Resolve UUIDs to semantic identifiers: + +```python +# ANTI-PATTERN: Returns raw UUIDs +def _run(self, query: str) -> str: + return json.dumps([{ + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "parent_id": "f1e2d3c4-b5a6-0987-dcba-654321098765" + }]) + +# PATTERN: Semantic identifiers +def _run(self, query: str) -> str: + return json.dumps([{ + "project_id": "PRJ-001", + "project_name": "Website Redesign", + "parent_project": "Marketing Initiatives" + }]) +``` + +## Principle 4: Meaningful Response Design + +### Curated Fields + +Return what enables action, exclude noise: + +```python +# ANTI-PATTERN: Dump everything +def _run(self, order_id: str) -> str: + order = db.get_order(order_id) + return json.dumps({ + "uuid": order.uuid, + "created_at_unix": order.created_at_unix, + "updated_at_unix": order.updated_at_unix, + "internal_flags": order.internal_flags, + "mime_type": order.mime_type, + "256px_image_url": order.thumbnail_url, + "customer_name": order.customer_name, + "status": order.status, + # ... 50 more fields + }) + +# PATTERN: Agent-relevant fields only +def _run(self, order_id: str) -> str: + order = db.get_order(order_id) + return json.dumps({ + "order_id": order.display_id, + "customer": order.customer_name, + "status": order.status, + "items": [{"name": i.name, "qty": i.qty} for i in order.items], + "total": f"${order.total:.2f}", + "can_modify": order.status in ["pending", "processing"] + }) +``` + +### Response Format Flexibility + +Let agents request appropriate detail levels: + +```python +class AnalysisInput(BaseModel): + target: str + response_format: str = Field( + default="concise", + description="'concise' (~50 tokens) or 'detailed' (~500 tokens)" + ) + +def _run(self, target: str, response_format: str = "concise") -> str: + analysis = self.analyze(target) + + if response_format == "concise": + return json.dumps({ + "summary": analysis.summary, + "score": analysis.score + }) + else: + return json.dumps({ + "summary": analysis.summary, + "score": analysis.score, + "breakdown": analysis.category_scores, + "evidence": analysis.supporting_data, + "methodology": analysis.methodology_notes + }) +``` + +## Principle 5: Actionable Error Messages + +### Replace Tracebacks with Guidance + +```python +# ANTI-PATTERN: Opaque errors +def _run(self, date: str) -> str: + try: + dt = datetime.fromisoformat(date) + except: + return "ValueError: Invalid isoformat string" + +# PATTERN: Corrective guidance +def _run(self, date: str) -> str: + try: + dt = datetime.fromisoformat(date) + except ValueError: + return ( + f"Invalid date format: '{date}'. " + f"Expected ISO 8601: YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS. " + f"Example: 2024-03-15 or 2024-03-15T14:30:00" + ) +``` + +### Suggest Alternatives + +```python +def _run(self, status: str) -> str: + valid_statuses = ["pending", "active", "completed", "cancelled"] + + if status not in valid_statuses: + # Find close matches + close = [s for s in valid_statuses if status.lower() in s] + suggestion = f" Did you mean '{close[0]}'?" if close else "" + return f"Invalid status: '{status}'.{suggestion} Valid options: {valid_statuses}" +``` + +### Guide Efficient Behavior + +```python +def _run(self, query: str, limit: int = 25) -> str: + results = db.search(query, limit=limit) + + if len(results) == limit: + return json.dumps({ + "results": results, + "note": f"Returned max {limit} results. Refine query for better matches." + }) + + return json.dumps({"results": results}) +``` + +## Principle 6: Effective Descriptions + +### Write for a New Team Member + +```python +description: str = """ +Search for customer support tickets by keyword or status. + +When to use: +- Finding tickets about a specific issue +- Checking status of customer-reported problems +- Looking up tickets by customer email + +Parameters: +- query: Search term (searches subject and body) +- status: Filter by 'open', 'pending', 'resolved', or 'all' +- assigned_to: Filter by agent email (optional) + +Returns: Ticket ID, subject, status, customer email, last update. +For full ticket details including conversation history, use 'get_ticket_details'. + +Note: Only returns last 30 days by default. Use date_range for older tickets. +""" +``` + +### Clarify Relationships + +```python +description: str = """ +Create a new task in a project. + +Requires: +- project_id: Get from 'search_projects' or 'list_my_projects' +- assignee_id: Get from 'search_team_members' (optional) + +Creates task and returns task_id for use with: +- 'add_task_comment': Add notes or updates +- 'update_task_status': Change status +- 'add_task_attachment': Attach files +""" +``` + +## Principle 7: Truncation and Limits + +### Default Limits + +```python +DEFAULT_LIMIT = 25 +MAX_RESPONSE_TOKENS = 25000 + +def _run(self, query: str, limit: int = DEFAULT_LIMIT) -> str: + results = db.search(query, limit=min(limit, 100)) + + response = json.dumps(results) + + # Enforce token limit + if len(response) > MAX_RESPONSE_TOKENS * 4: # ~4 chars per token + return json.dumps({ + "results": results[:10], + "truncated": True, + "total_available": len(results), + "suggestion": "Refine your query for more specific results" + }) + + return response +``` + +### Pagination Guidance + +```python +def _run(self, query: str, page: int = 1) -> str: + per_page = 25 + total, results = db.search_paginated(query, page, per_page) + + return json.dumps({ + "results": results, + "page": page, + "total_pages": (total + per_page - 1) // per_page, + "hint": "Use 'page' parameter for more results" if total > per_page else None + }) +``` + +## Principle 8: Evaluation-Driven Development + +### Build Real-World Evaluations + +Test tools with realistic, multi-step scenarios: + +```python +# Evaluation scenarios +EVAL_SCENARIOS = [ + { + "name": "customer_lookup_flow", + "prompt": "Find John Smith's last order and check if it shipped", + "expected_tools": ["search_customers", "get_order_details"], + "success_criteria": lambda result: "shipped" in result.lower() or "pending" in result.lower() + }, + { + "name": "refund_processing", + "prompt": "Process a refund for order ORD-12345", + "expected_tools": ["get_order_details", "process_refund"], + "success_criteria": lambda result: "refund" in result.lower() and "processed" in result.lower() + } +] +``` + +### Metrics to Track + +- **Accuracy**: Did the agent complete the task correctly? +- **Tool calls**: How many calls were needed? (fewer is better) +- **Token consumption**: Total tokens used +- **Error rate**: How often did tools return errors? +- **Recovery rate**: Did the agent recover from errors? + +### Iterate with Agent Feedback + +After evaluations, let agents analyze their own transcripts: + +``` +Analyze this tool usage transcript. Identify: +1. Where did I get confused about which tool to use? +2. Which tool descriptions were unclear? +3. What information was missing from tool responses? +4. Where did I make unnecessary tool calls? +``` + +## Summary Checklist + +Before deploying a tool, verify: + +- [ ] **Context Efficient**: Returns only necessary data +- [ ] **Clear Purpose**: Distinct from other tools, unambiguous name +- [ ] **Semantic Names**: Parameters are self-documenting +- [ ] **Meaningful Responses**: Curated fields, no noise +- [ ] **Actionable Errors**: Guide correction, suggest alternatives +- [ ] **Good Description**: Written for a new team member +- [ ] **Reasonable Limits**: Default pagination, token limits +- [ ] **Tested**: Evaluated with realistic multi-step scenarios diff --git a/.claude/skills/crewai-tool-creator/references/tool-patterns.md b/.claude/skills/crewai-tool-creator/references/tool-patterns.md new file mode 100644 index 000000000..fca406e00 --- /dev/null +++ b/.claude/skills/crewai-tool-creator/references/tool-patterns.md @@ -0,0 +1,600 @@ +# Advanced Tool Patterns + +## Tool Composition + +Create tools that work together as a cohesive toolkit: + +```python +from typing import Type +from crewai.tools import BaseTool +from pydantic import BaseModel, Field + +# Shared connection manager +class DatabaseToolMixin: + """Mixin for tools that need database access.""" + + def __init__(self, db_connection): + super().__init__() + self.db = db_connection + +# Tool 1: Search +class ProjectSearchInput(BaseModel): + query: str = Field(..., min_length=2) + status: str = Field(default="all") + +class ProjectSearchTool(DatabaseToolMixin, BaseTool): + name: str = "project_search" + description: str = """ + Search for projects by name or description. + Returns project_id, name, status, and owner. + Use project_id with 'project_get_details' for full info. + """ + args_schema: Type[BaseModel] = ProjectSearchInput + + def _run(self, query: str, status: str = "all") -> str: + results = self.db.search_projects(query, status) + return json.dumps([{ + "project_id": p.id, + "name": p.name, + "status": p.status, + "owner": p.owner_name + } for p in results]) + +# Tool 2: Get Details (uses project_id from search) +class ProjectDetailsInput(BaseModel): + project_id: str = Field(..., description="Project ID from search results") + +class ProjectDetailsTool(DatabaseToolMixin, BaseTool): + name: str = "project_get_details" + description: str = """ + Get full project details by project_id. + Use after 'project_search' to get comprehensive information. + """ + args_schema: Type[BaseModel] = ProjectDetailsInput + + def _run(self, project_id: str) -> str: + project = self.db.get_project(project_id) + if not project: + return f"Project '{project_id}' not found. Use 'project_search' to find valid IDs." + return json.dumps({ + "project_id": project.id, + "name": project.name, + "description": project.description, + "status": project.status, + "owner": project.owner_name, + "team": [m.name for m in project.team], + "milestones": [{ + "name": m.name, + "due": m.due_date.isoformat(), + "complete": m.is_complete + } for m in project.milestones] + }) + +# Tool 3: Update (uses project_id) +class ProjectUpdateInput(BaseModel): + project_id: str = Field(...) + status: str = Field(default=None) + description: str = Field(default=None) + +class ProjectUpdateTool(DatabaseToolMixin, BaseTool): + name: str = "project_update" + description: str = """ + Update project status or description. + Requires project_id from search. At least one field must be provided. + """ + args_schema: Type[BaseModel] = ProjectUpdateInput + + def _run(self, project_id: str, status: str = None, description: str = None) -> str: + if not status and not description: + return "No updates provided. Specify 'status' and/or 'description'." + + updates = {} + if status: + valid = ["active", "paused", "completed", "cancelled"] + if status not in valid: + return f"Invalid status. Choose from: {valid}" + updates["status"] = status + if description: + updates["description"] = description + + self.db.update_project(project_id, **updates) + return f"Project {project_id} updated successfully." +``` + +## Stateful Tools + +Tools that maintain state across calls: + +```python +class ConversationMemoryTool(BaseTool): + name: str = "conversation_memory" + description: str = """ + Store and retrieve conversation context. + Actions: 'store' to save, 'recall' to retrieve, 'clear' to reset. + """ + args_schema: Type[BaseModel] = MemoryInput + + def __init__(self): + super().__init__() + self._memory: dict[str, list] = {} + + def _run(self, action: str, key: str, value: str = None) -> str: + if action == "store": + if key not in self._memory: + self._memory[key] = [] + self._memory[key].append({ + "value": value, + "timestamp": datetime.now().isoformat() + }) + return f"Stored under '{key}'" + + elif action == "recall": + items = self._memory.get(key, []) + if not items: + return f"No memory found for '{key}'" + return json.dumps(items) + + elif action == "clear": + if key in self._memory: + del self._memory[key] + return f"Cleared '{key}'" + return f"Nothing to clear for '{key}'" + + return f"Unknown action: {action}. Use 'store', 'recall', or 'clear'." +``` + +## Retry and Circuit Breaker Patterns + +```python +from functools import wraps +import time + +class ResilientAPITool(BaseTool): + name: str = "api_call" + description: str = "Make API calls with automatic retry" + args_schema: Type[BaseModel] = APIInput + + def __init__(self, api_client, max_retries: int = 3): + super().__init__() + self.api = api_client + self.max_retries = max_retries + self._failures = 0 + self._circuit_open_until = None + + def _run(self, endpoint: str, params: dict = None) -> str: + # Circuit breaker check + if self._circuit_open_until: + if datetime.now() < self._circuit_open_until: + return "Service temporarily unavailable. Try again in 60 seconds." + self._circuit_open_until = None + self._failures = 0 + + last_error = None + for attempt in range(self.max_retries): + try: + result = self.api.call(endpoint, params) + self._failures = 0 # Reset on success + return json.dumps(result) + except APIError as e: + last_error = e + if attempt < self.max_retries - 1: + time.sleep(2 ** attempt) # Exponential backoff + + # Track failures for circuit breaker + self._failures += 1 + if self._failures >= 5: + self._circuit_open_until = datetime.now() + timedelta(seconds=60) + + return f"API call failed after {self.max_retries} attempts: {last_error}" +``` + +## Paginated Results Pattern + +```python +class PaginatedSearchInput(BaseModel): + query: str = Field(...) + page: int = Field(default=1, ge=1) + per_page: int = Field(default=25, ge=1, le=100) + +class PaginatedSearchTool(BaseTool): + name: str = "paginated_search" + description: str = """ + Search with pagination. Returns results and pagination info. + Use 'page' parameter to navigate through results. + """ + args_schema: Type[BaseModel] = PaginatedSearchInput + + def _run(self, query: str, page: int = 1, per_page: int = 25) -> str: + total, results = self.db.search_paginated(query, page, per_page) + total_pages = (total + per_page - 1) // per_page + + return json.dumps({ + "results": results, + "pagination": { + "current_page": page, + "per_page": per_page, + "total_results": total, + "total_pages": total_pages, + "has_next": page < total_pages, + "has_prev": page > 1 + } + }) +``` + +## File Operation Tools + +```python +import os +from pathlib import Path + +class FileReadInput(BaseModel): + file_path: str = Field(..., description="Path relative to workspace root") + encoding: str = Field(default="utf-8") + +class SafeFileReadTool(BaseTool): + name: str = "read_file" + description: str = """ + Read file contents from the workspace. + Only files within the workspace directory can be accessed. + """ + args_schema: Type[BaseModel] = FileReadInput + + def __init__(self, workspace_root: str): + super().__init__() + self.workspace = Path(workspace_root).resolve() + + def _run(self, file_path: str, encoding: str = "utf-8") -> str: + # Security: prevent path traversal + target = (self.workspace / file_path).resolve() + if not str(target).startswith(str(self.workspace)): + return "Error: Access denied. Path must be within workspace." + + if not target.exists(): + return f"File not found: {file_path}" + + if not target.is_file(): + return f"Not a file: {file_path}" + + # Size limit + if target.stat().st_size > 1_000_000: # 1MB + return "File too large (>1MB). Use 'read_file_chunk' for large files." + + try: + return target.read_text(encoding=encoding) + except UnicodeDecodeError: + return f"Cannot read file with encoding '{encoding}'. Try 'latin-1' or 'binary'." +``` + +## Tool Testing Patterns + +```python +import pytest +from unittest.mock import Mock, patch + +class TestCustomerSearchTool: + """Test suite for CustomerSearchTool.""" + + @pytest.fixture + def mock_db(self): + db = Mock() + db.search_customers.return_value = [ + Mock(id="C001", full_name="John Doe", email="john@example.com", + status="active", last_order_date=None) + ] + return db + + @pytest.fixture + def tool(self, mock_db): + return CustomerSearchTool(db_connection=mock_db) + + def test_basic_search(self, tool): + result = tool._run(query="john") + data = json.loads(result) + assert data["count"] == 1 + assert data["customers"][0]["name"] == "John Doe" + + def test_no_results(self, tool, mock_db): + mock_db.search_customers.return_value = [] + result = tool._run(query="nonexistent") + assert "No customers found" in result + + def test_invalid_status(self, tool): + result = tool._run(query="john", status="invalid") + assert "Invalid status" in result + + def test_db_error_handling(self, tool, mock_db): + mock_db.search_customers.side_effect = DatabaseError("Connection lost") + result = tool._run(query="john") + assert "Database unavailable" in result +``` + +## Structured Tool Output with Pydantic + +```python +class AnalysisOutput(BaseModel): + """Structured output for analysis tool.""" + summary: str + confidence: float = Field(ge=0, le=1) + key_findings: list[str] + recommendations: list[str] + +class AnalysisTool(BaseTool): + name: str = "analyze_data" + description: str = "Analyze data and return structured findings" + args_schema: Type[BaseModel] = AnalysisInput + + def _run(self, data_source: str) -> str: + raw_analysis = self.analyzer.analyze(data_source) + + # Structure the output + output = AnalysisOutput( + summary=raw_analysis.summary, + confidence=raw_analysis.confidence_score, + key_findings=raw_analysis.findings[:5], + recommendations=raw_analysis.recommendations[:3] + ) + + return output.model_dump_json() +``` + +## Tool Initialization in Crews + +```python +# In your crew file +from crewai import Agent, Crew, Task +from crewai.project import CrewBase, agent, crew, task +from ..tools.customer_tools import CustomerSearchTool, CustomerDetailsTool +from ..tools.order_tools import OrderSearchTool + +@CrewBase +class SupportCrew: + agents_config = "config/agents.yaml" + tasks_config = "config/tasks.yaml" + + def __init__(self, db_connection): + super().__init__() + # Initialize tools with shared dependencies + self.customer_search = CustomerSearchTool(db_connection) + self.customer_details = CustomerDetailsTool(db_connection) + self.order_search = OrderSearchTool(db_connection) + + @agent + def support_agent(self) -> Agent: + return Agent( + config=self.agents_config["support_agent"], # type: ignore[index] + tools=[ + self.customer_search, + self.customer_details, + self.order_search + ], + verbose=True, + ) + + @task + def resolve_inquiry(self) -> Task: + return Task(config=self.tasks_config["resolve_inquiry"]) # type: ignore[index] + + @crew + def crew(self) -> Crew: + return Crew(agents=self.agents, tasks=self.tasks) +``` + +## Tool Caching + +CrewAI caches tool results to avoid redundant calls. Use `cache_function` for fine-grained control. + +### Cache Function Signature + +```python +def cache_function(arguments: dict, result: Any) -> bool: + """ + Determines whether to cache a result. + + Args: + arguments: Dict of arguments passed to _run() + result: The return value from _run() + + Returns: + True to cache, False to skip caching + """ + return True +``` + +### Always Cache (Default Behavior) + +```python +from typing import Callable, Any + +class ExpensiveAPITool(BaseTool): + name: str = "expensive_api" + description: str = "Call expensive API with caching" + args_schema: Type[BaseModel] = APIInput + + # Cache all results + cache_function: Callable[[dict, Any], bool] = lambda args, result: True + + def _run(self, query: str) -> str: + return expensive_api.call(query) +``` + +### Conditional Caching - Only Cache Successes + +```python +class DataFetchTool(BaseTool): + name: str = "fetch_data" + description: str = "Fetch data, only cache successful results" + args_schema: Type[BaseModel] = FetchInput + + def _should_cache(self, args: dict, result: Any) -> bool: + """Don't cache error responses.""" + if isinstance(result, str): + return not result.startswith("Error:") + return True + + cache_function: Callable[[dict, Any], bool] = _should_cache + + def _run(self, data_id: str) -> str: + try: + data = api.fetch(data_id) + return json.dumps(data) + except APIError as e: + return f"Error: {e}" # Won't be cached, agent can retry +``` + +### Conditional Caching - Based on Result Value + +```python +class CalculationTool(BaseTool): + name: str = "calculate" + description: str = "Calculate with selective caching" + args_schema: Type[BaseModel] = CalcInput + + # Only cache positive results + cache_function: Callable[[dict, Any], bool] = lambda args, result: ( + isinstance(result, (int, float)) and result > 0 + ) + + def _run(self, a: int, b: int) -> int: + return a * b +``` + +### Conditional Caching - Based on Arguments + +```python +class SearchTool(BaseTool): + name: str = "search" + description: str = "Search with argument-based caching" + args_schema: Type[BaseModel] = SearchInput + + def _cache_strategy(self, args: dict, result: Any) -> bool: + """Cache only broad searches, not user-specific ones.""" + # Don't cache user-specific searches (may change frequently) + if args.get("user_id"): + return False + # Cache general searches + return True + + cache_function: Callable[[dict, Any], bool] = _cache_strategy + + def _run(self, query: str, user_id: str = None) -> str: + return db.search(query, user_id=user_id) +``` + +## Async Execution Patterns + +**Important:** CrewAI tool `_run()` methods are synchronous. Async execution happens at the crew/task level. + +### Tools Are Synchronous + +```python +import requests + +class FetchURLTool(BaseTool): + name: str = "fetch_url" + description: str = "Fetch content from URL" + args_schema: Type[BaseModel] = URLInput + + def _run(self, url: str, timeout: int = 30) -> str: + """Use synchronous HTTP client in _run().""" + try: + response = requests.get(url, timeout=timeout) + response.raise_for_status() + return response.text + except requests.Timeout: + return f"Timeout after {timeout}s. Try increasing timeout." + except requests.RequestException as e: + return f"Request failed: {e}" +``` + +### Parallel External Calls Within a Tool + +When a tool needs multiple independent external calls, use `ThreadPoolExecutor`: + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +class MultiSourceSearchTool(BaseTool): + name: str = "multi_source_search" + description: str = "Search multiple sources in parallel" + args_schema: Type[BaseModel] = SearchInput + + def __init__(self, sources: list): + super().__init__() + self.sources = sources + + def _run(self, query: str) -> str: + with ThreadPoolExecutor(max_workers=len(self.sources)) as executor: + futures = { + executor.submit(src.search, query): src.name + for src in self.sources + } + + results = {} + for future in as_completed(futures): + source_name = futures[future] + try: + results[source_name] = future.result(timeout=10) + except Exception as e: + results[source_name] = f"Error: {e}" + + return json.dumps(results) +``` + +### Async at Crew Level + +Async execution is configured when kicking off crews: + +```python +from crewai.flow.flow import Flow, listen + +class ResearchFlow(Flow[ResearchState]): + + @listen(classify_topic) + async def run_research_crew(self): + crew = ResearchCrew().crew() + + # Option 1: Native async (preferred for high concurrency) + result = await crew.akickoff(inputs={"topic": self.state.topic}) + + # Option 2: Thread-based async + result = await crew.kickoff_async(inputs={"topic": self.state.topic}) + + self.state.research = result.raw + return result + + @listen(run_research_crew) + async def run_parallel_crews(self): + # Run multiple crews concurrently + crews = [ + AnalysisCrew().crew(), + SummaryCrew().crew(), + ValidationCrew().crew() + ] + + results = await asyncio.gather(*[ + crew.akickoff(inputs={"data": self.state.research}) + for crew in crews + ]) + + return results +``` + +### Async Task Configuration + +Mark tasks for async execution within a crew: + +```python +# config/tasks.yaml +research_task: + description: "Research the topic thoroughly" + expected_output: "Comprehensive research findings" + agent: researcher + async_execution: true # This task runs asynchronously + +analysis_task: + description: "Analyze research findings" + expected_output: "Analysis report" + agent: analyst + context: + - research_task # Waits for research_task to complete +``` diff --git a/.claude/skills/software-architect/SKILL.md b/.claude/skills/software-architect/SKILL.md new file mode 100644 index 000000000..ff10b25e7 --- /dev/null +++ b/.claude/skills/software-architect/SKILL.md @@ -0,0 +1,419 @@ +--- +name: software-architect +description: | + Guide for writing clean, maintainable code following SOLID principles. + Use this skill when: (1) Writing a function that does multiple things (validate, save, notify), + (2) Adding if/elif chains for new feature variations, (3) Creating classes with inheritance, + (4) A class/function requires dependencies it doesn't fully use, (5) Code is hard to test + because it creates its own dependencies, (6) Refactoring for better structure. + Apply these principles to ensure code is easy to understand, modify, and test. +--- + +# Clean Code with SOLID Principles + +**Core Philosophy: Write code that is easy to understand, change, and test.** + +SOLID is an acronym for five principles that help you write better code. Apply these every time you write functions or classes. + +## Quick Checklist Before Writing Code + +| Question to Ask | If Yes, You're Good | If No, Refactor | +|-----------------|---------------------|-----------------| +| Does this function/class do ONE thing? | ✓ | Split it up | +| Can I add features without changing existing code? | ✓ | Use abstractions | +| Can I replace this with a similar component? | ✓ | Fix the contract | +| Am I only using what I need? | ✓ | Create smaller interfaces | +| Do I depend on abstractions, not specifics? | ✓ | Inject dependencies | + +--- + +## S - Single Responsibility Principle + +**"A function or class should do one thing and do it well."** + +### Why It Matters +- Easier to understand (one purpose = one mental model) +- Easier to test (test one thing at a time) +- Easier to change (change one thing without breaking others) + +### Bad Example + +```python +def process_user_registration(email: str, password: str) -> dict: + # Validates email + if "@" not in email: + raise ValueError("Invalid email") + + # Validates password + if len(password) < 8: + raise ValueError("Password too short") + + # Creates user in database + user_id = database.insert("users", {"email": email, "password": hash(password)}) + + # Sends welcome email + smtp.send(email, "Welcome!", "Thanks for joining!") + + # Logs the registration + logger.info(f"New user registered: {email}") + + return {"user_id": user_id, "email": email} +``` + +**Problem**: This function does 5 different things. If you need to change how emails are sent, you're touching the same code that handles validation and database operations. + +### Good Example + +```python +def validate_email(email: str) -> bool: + """Check if email format is valid.""" + return "@" in email and "." in email + +def validate_password(password: str) -> bool: + """Check if password meets requirements.""" + return len(password) >= 8 + +def create_user(email: str, password: str) -> str: + """Create user in database and return user ID.""" + return database.insert("users", {"email": email, "password": hash(password)}) + +def send_welcome_email(email: str) -> None: + """Send welcome email to new user.""" + smtp.send(email, "Welcome!", "Thanks for joining!") + +def register_user(email: str, password: str) -> dict: + """Orchestrate the user registration process.""" + if not validate_email(email): + raise ValueError("Invalid email") + if not validate_password(password): + raise ValueError("Password too short") + + user_id = create_user(email, password) + send_welcome_email(email) + + return {"user_id": user_id, "email": email} +``` + +**Benefits**: +- Each function is easy to understand +- You can test `validate_email` without a database +- You can change email sending without touching validation + +--- + +## O - Open/Closed Principle + +**"Code should be open for extension but closed for modification."** + +### Why It Matters +- Add new features without changing existing code +- Reduces risk of breaking things that already work +- Makes your code more flexible + +### Bad Example + +```python +def calculate_discount(customer_type: str, amount: float) -> float: + """Calculate discount based on customer type.""" + if customer_type == "regular": + return amount * 0.05 + elif customer_type == "premium": + return amount * 0.10 + elif customer_type == "vip": + return amount * 0.20 + else: + return 0.0 + +# Problem: To add a new customer type, you MUST modify this function +# What if you forget a case? What if this function is used everywhere? +``` + +### Good Example + +```python +# Define discount strategies +DISCOUNT_RATES = { + "regular": 0.05, + "premium": 0.10, + "vip": 0.20, +} + +def calculate_discount(customer_type: str, amount: float) -> float: + """Calculate discount based on customer type.""" + rate = DISCOUNT_RATES.get(customer_type, 0.0) + return amount * rate + +# To add a new customer type, just add to the dictionary: +# DISCOUNT_RATES["enterprise"] = 0.25 +# No need to modify the function! +``` + +--- + +## L - Liskov Substitution Principle + +**"If you replace a parent with a child, things should still work."** + +### Why It Matters +- Ensures your code is truly reusable +- Prevents unexpected bugs when using inheritance +- Makes your class hierarchies trustworthy + +### Bad Example + +```python +class Bird: + def fly(self) -> str: + return "Flying high!" + +class Penguin(Bird): + def fly(self) -> str: + raise Exception("Penguins can't fly!") # BREAKS the contract! + +def make_bird_fly(bird: Bird) -> str: + return bird.fly() + +# This will crash unexpectedly: +penguin = Penguin() +make_bird_fly(penguin) # Exception: Penguins can't fly! +``` + +**Problem**: `Penguin` inherits from `Bird` but can't fulfill the `fly()` contract. Code expecting a `Bird` will break. + +### Good Example + +```python +class Bird: + def move(self) -> str: + return "Moving" + +class FlyingBird(Bird): + def fly(self) -> str: + return "Flying high!" + +class SwimmingBird(Bird): + def swim(self) -> str: + return "Swimming!" + +class Eagle(FlyingBird): + def fly(self) -> str: + return "Soaring through the sky!" + +class Penguin(SwimmingBird): + def swim(self) -> str: + return "Swimming gracefully!" + +# Now each bird type can be used correctly: +def make_bird_fly(bird: FlyingBird) -> str: + return bird.fly() + +def make_bird_swim(bird: SwimmingBird) -> str: + return bird.swim() + +eagle = Eagle() +make_bird_fly(eagle) # Works! + +penguin = Penguin() +make_bird_swim(penguin) # Works! +``` + +### Simple Rule +If your child class needs to throw an exception or return `None` for a method that the parent defines, you probably have the wrong inheritance structure. + +--- + +## I - Interface Segregation Principle + +**"Don't force code to depend on things it doesn't use."** + +### Why It Matters +- Keeps your code focused and lean +- Reduces unnecessary dependencies +- Makes testing easier + +### Bad Example + +```python +class Worker: + def work(self) -> str: + pass + + def eat(self) -> str: + pass + + def sleep(self) -> str: + pass + +class Robot(Worker): + def work(self) -> str: + return "Working..." + + def eat(self) -> str: + raise Exception("Robots don't eat!") # Forced to implement this! + + def sleep(self) -> str: + raise Exception("Robots don't sleep!") # Forced to implement this! +``` + +**Problem**: `Robot` is forced to implement `eat()` and `sleep()` even though it doesn't need them. + +### Good Example + +```python +class Workable: + def work(self) -> str: + pass + +class Eatable: + def eat(self) -> str: + pass + +class Sleepable: + def sleep(self) -> str: + pass + +class Human(Workable, Eatable, Sleepable): + def work(self) -> str: + return "Working..." + + def eat(self) -> str: + return "Eating lunch..." + + def sleep(self) -> str: + return "Sleeping..." + +class Robot(Workable): # Only implements what it needs! + def work(self) -> str: + return "Working 24/7..." +``` + +### Practical Application: Function Parameters + +```python +# Bad: Function takes more than it needs +def send_notification(user: User) -> None: + # Only uses user.email, but requires entire User object + email_service.send(user.email, "Hello!") + +# Good: Function takes only what it needs +def send_notification(email: str) -> None: + email_service.send(email, "Hello!") + +# Now you can call it without having a full User object: +send_notification("user@example.com") +``` + +--- + +## D - Dependency Inversion Principle + +**"Depend on abstractions, not concrete implementations."** + +### Why It Matters +- Makes code flexible and swappable +- Makes testing much easier (use fakes/mocks) +- Reduces coupling between components + +### Bad Example + +```python +class EmailService: + def send(self, to: str, message: str) -> None: + # Sends email via SMTP + smtp_server.send(to, message) + +class UserRegistration: + def __init__(self): + self.email_service = EmailService() # HARD-CODED dependency! + + def register(self, email: str, password: str) -> None: + # Create user... + user_id = create_user(email, password) + # Send welcome email + self.email_service.send(email, "Welcome!") + +# Problem: Can't test without actually sending emails! +# Problem: Can't switch to a different email provider easily +``` + +### Good Example + +```python +from abc import ABC, abstractmethod + +# 1. Define what you need (abstraction) +class NotificationService(ABC): + @abstractmethod + def send(self, to: str, message: str) -> None: + pass + +# 2. Create implementations +class EmailNotificationService(NotificationService): + def send(self, to: str, message: str) -> None: + smtp_server.send(to, message) + +class SMSNotificationService(NotificationService): + def send(self, to: str, message: str) -> None: + sms_gateway.send(to, message) + +# 3. Depend on the abstraction, not the implementation +class UserRegistration: + def __init__(self, notification_service: NotificationService): + self.notification_service = notification_service # INJECTED! + + def register(self, email: str, password: str) -> None: + user_id = create_user(email, password) + self.notification_service.send(email, "Welcome!") + +# Usage - you choose which implementation to use: +email_service = EmailNotificationService() +registration = UserRegistration(email_service) + +# For testing - use a fake: +class FakeNotificationService(NotificationService): + def __init__(self): + self.sent_messages = [] + + def send(self, to: str, message: str) -> None: + self.sent_messages.append((to, message)) + +# Test without sending real emails: +fake_service = FakeNotificationService() +registration = UserRegistration(fake_service) +registration.register("test@example.com", "password123") +assert len(fake_service.sent_messages) == 1 +``` + +--- + +## Quick Function Guidelines + +| Guideline | Recommendation | +|-----------|----------------| +| **Length** | Keep functions under 20 lines | +| **Arguments** | Prefer 0-3 arguments | +| **Naming** | Use verb + noun: `calculate_total`, `send_email` | +| **One thing** | Each function does exactly one thing | +| **No surprises** | Function does what its name says, nothing more | + +--- + +## Summary: SOLID at a Glance + +| Principle | In Simple Terms | Quick Test | +|-----------|----------------|------------| +| **S**ingle Responsibility | One function = one job | Can you describe it without saying "and"? | +| **O**pen/Closed | Add features, don't modify | Can you extend without editing? | +| **L**iskov Substitution | Children honor parent's promises | Does every child work where parent works? | +| **I**nterface Segregation | Don't force unused dependencies | Is everything you require actually used? | +| **D**ependency Inversion | Depend on abstractions | Can you swap implementations easily? | + +--- + +## When to Apply + +- **Always apply S**: Every function should do one thing +- **Apply O when**: You find yourself adding `if/elif` chains for new cases +- **Apply L when**: Using inheritance +- **Apply I when**: Your interfaces have methods some implementers don't need +- **Apply D when**: You want testable code or need flexibility diff --git a/.claude/skills/streamlit/SKILL.md b/.claude/skills/streamlit/SKILL.md new file mode 100644 index 000000000..5d2488c5f --- /dev/null +++ b/.claude/skills/streamlit/SKILL.md @@ -0,0 +1,333 @@ +--- +name: streamlit +description: | + Guide for building Streamlit web applications in Python. Use this skill when: + - Creating interactive data apps, dashboards, or web UIs with Python + - Working with st.* functions for widgets, layouts, charts, or data display + - Implementing caching (@st.cache_data, @st.cache_resource) or session state + - Building multipage Streamlit apps with st.navigation + - Configuring Streamlit themes, secrets, or database connections +--- + +# Streamlit Development Guide + +Streamlit turns Python scripts into interactive web apps. The framework reruns your entire script from top to bottom whenever users interact with widgets or source code changes. + +## Core Data Flow + +```python +import streamlit as st + +# Script runs top-to-bottom on every interaction +x = st.slider('Value', 0, 100, 50) # Widget interaction triggers rerun +st.write(f"Result: {x * 2}") +``` + +**Key Principle**: Every widget interaction causes a full script rerun. Use caching and session state to preserve expensive computations and user data. + +## Essential Patterns + +### Display Data + +```python +import streamlit as st +import pandas as pd + +# Magic: standalone variables auto-render +df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) +df # Automatically displayed + +# Explicit display +st.write("Swiss Army knife for any data type") +st.dataframe(df.style.highlight_max(axis=0)) # Interactive +st.table(df) # Static +``` + +### Widgets with Keys + +```python +import streamlit as st + +# Access widget values via session_state +st.text_input("Name", key="user_name") +st.number_input("Age", key="user_age") + +# Use values anywhere +if st.session_state.user_name: + st.write(f"Hello {st.session_state.user_name}!") +``` + +### Layout + +```python +import streamlit as st + +# Sidebar +with st.sidebar: + option = st.selectbox("Choose", ["A", "B", "C"]) + threshold = st.slider("Threshold", 0, 100) + +# Columns +col1, col2, col3 = st.columns(3) +with col1: + st.metric("Metric 1", 42) +with col2: + st.metric("Metric 2", 84) +with col3: + st.metric("Metric 3", 126) + +# Tabs +tab1, tab2 = st.tabs(["Chart", "Data"]) +with tab1: + st.line_chart(data) +with tab2: + st.dataframe(data) +``` + +### Caching (Critical for Performance) + +```python +import streamlit as st + +# Cache data computations (returns copy each call) +@st.cache_data +def load_data(path: str) -> pd.DataFrame: + return pd.read_csv(path) + +# Cache resources (returns same object, shared across sessions) +@st.cache_resource +def load_model(name: str): + return SomeLargeModel(name) + +# Usage +df = load_data("data.csv") # Cached after first call +model = load_model("gpt") # Shared across all users +``` + +**Rule of Thumb**: +- `@st.cache_data` → DataFrames, dicts, lists, strings (serializable data) +- `@st.cache_resource` → ML models, database connections (global resources) + +### Session State + +```python +import streamlit as st + +# Initialize state (only runs once per session) +if "counter" not in st.session_state: + st.session_state.counter = 0 + st.session_state.history = [] + +# Modify state +if st.button("Increment"): + st.session_state.counter += 1 + st.session_state.history.append(st.session_state.counter) + +st.write(f"Count: {st.session_state.counter}") +``` + +### Database Connections + +```python +import streamlit as st + +# Automatic connection management with caching +conn = st.connection("my_database") +df = conn.query("SELECT * FROM users WHERE active = true") +st.dataframe(df) +``` + +Configure in `.streamlit/secrets.toml`: +```toml +[connections.my_database] +type = "sql" +dialect = "postgresql" +host = "localhost" +port = 5432 +database = "mydb" +username = "user" +password = "pass" +``` + +### Multipage Apps + +```python +# streamlit_app.py (entry point) +import streamlit as st + +# Define pages +home = st.Page("pages/home.py", title="Home", icon="🏠") +dashboard = st.Page("pages/dashboard.py", title="Dashboard", icon="📊") +settings = st.Page("pages/settings.py", title="Settings", icon="⚙️") + +# Create navigation +pg = st.navigation([home, dashboard, settings]) +pg.run() +``` + +Project structure: +``` +my_app/ +├── streamlit_app.py # Entry point with navigation +├── pages/ +│ ├── home.py +│ ├── dashboard.py +│ └── settings.py +└── .streamlit/ + ├── config.toml # Theme configuration + └── secrets.toml # Database credentials (gitignored) +``` + +## Charts and Visualization + +```python +import streamlit as st +import pandas as pd +import numpy as np + +# Built-in charts +st.line_chart(df) +st.bar_chart(df) +st.area_chart(df) +st.scatter_chart(df, x="col1", y="col2", color="category") + +# Maps +st.map(df) # Requires 'lat' and 'lon' columns + +# External libraries +import plotly.express as px +fig = px.scatter(df, x="x", y="y", color="category") +st.plotly_chart(fig) + +import altair as alt +chart = alt.Chart(df).mark_circle().encode(x='x', y='y') +st.altair_chart(chart) +``` + +## Forms and Callbacks + +```python +import streamlit as st + +# Forms batch inputs (single rerun on submit) +with st.form("my_form"): + name = st.text_input("Name") + email = st.text_input("Email") + submitted = st.form_submit_button("Submit") + +if submitted: + st.success(f"Registered {name} with {email}") + +# Callbacks execute before script reruns +def on_change(): + st.session_state.processed = process(st.session_state.input_value) + +st.text_input("Input", key="input_value", on_change=on_change) +``` + +## Progress and Status + +```python +import streamlit as st +import time + +# Progress bar +progress = st.progress(0) +for i in range(100): + progress.progress(i + 1) + time.sleep(0.01) + +# Status messages +st.success("Operation completed!") +st.error("Something went wrong") +st.warning("Check your input") +st.info("Processing...") + +# Spinner +with st.spinner("Loading..."): + time.sleep(2) +st.success("Done!") + +# Empty placeholder for dynamic updates +placeholder = st.empty() +placeholder.text("Waiting...") +# Later... +placeholder.text("Updated!") +``` + +## File Handling + +```python +import streamlit as st + +# File upload +uploaded = st.file_uploader("Choose a file", type=["csv", "xlsx"]) +if uploaded: + df = pd.read_csv(uploaded) + st.dataframe(df) + +# File download +csv = df.to_csv(index=False) +st.download_button( + label="Download CSV", + data=csv, + file_name="data.csv", + mime="text/csv" +) +``` + +## Theming + +Configure in `.streamlit/config.toml`: +```toml +[theme] +primaryColor = "#FF4B4B" +backgroundColor = "#FFFFFF" +secondaryBackgroundColor = "#F0F2F6" +textColor = "#262730" +font = "sans serif" +``` + +## Static Files + +For direct URL access to files: +``` +my_app/ +├── static/ +│ └── logo.png # Accessible at /app/static/logo.png +└── streamlit_app.py +``` + +Enable in config: +```toml +[server] +enableStaticServing = true +``` + +## Running Apps + +```bash +# Development +streamlit run app.py + +# With arguments +streamlit run app.py -- --data-path ./data + +# Configuration +streamlit run app.py --server.port 8080 --server.headless true +``` + +## Best Practices + +1. **Minimize reruns**: Use `@st.cache_data` for expensive operations +2. **Preserve state**: Use `st.session_state` for user data that should persist +3. **Batch inputs**: Use `st.form` when multiple inputs should submit together +4. **Structure large apps**: Use multipage navigation with `st.Page` and `st.navigation` +5. **Secure secrets**: Store credentials in `.streamlit/secrets.toml`, never in code +6. **Responsive layout**: Use `st.columns` and `st.sidebar` for organized UIs + +## Reference Files + +- [references/widgets-catalog.md](references/widgets-catalog.md) - Complete widget reference: text inputs, selection, numeric, date/time, buttons, callbacks +- [references/layout-patterns.md](references/layout-patterns.md) - Advanced layouts: columns, containers, tabs, dialogs, dashboard examples +- [references/state-patterns.md](references/state-patterns.md) - State patterns: forms, multi-step wizards, authentication, cache vs session state diff --git a/.claude/skills/streamlit/references/layout-patterns.md b/.claude/skills/streamlit/references/layout-patterns.md new file mode 100644 index 000000000..9c548ce01 --- /dev/null +++ b/.claude/skills/streamlit/references/layout-patterns.md @@ -0,0 +1,222 @@ +# Streamlit Layout Patterns + +Advanced layout techniques for building professional Streamlit apps. + +## Contents + +- [Column Layouts](#column-layouts) +- [Container Patterns](#container-patterns) +- [Expander](#expander) +- [Tabs](#tabs) +- [Sidebar](#sidebar) +- [Empty Placeholders](#empty-placeholders) +- [Popover](#popover) +- [Dialog (Modal)](#dialog-modal) +- [Page Configuration](#page-configuration) +- [Dashboard Layout Example](#dashboard-layout-example) + +## Column Layouts + +```python +import streamlit as st + +# Equal columns +col1, col2, col3 = st.columns(3) + +# Weighted columns +left, right = st.columns([2, 1]) # 2:1 ratio + +# With gap control +cols = st.columns(3, gap="large") # small, medium, large + +# Nested columns +outer1, outer2 = st.columns(2) +with outer1: + inner1, inner2 = st.columns(2) + with inner1: + st.metric("A", 100) + with inner2: + st.metric("B", 200) +``` + +## Container Patterns + +```python +import streamlit as st + +# Basic container +with st.container(): + st.write("Grouped content") + +# Container with border +with st.container(border=True): + st.write("Boxed content") + +# Fixed height with scrolling +with st.container(height=300): + for i in range(50): + st.write(f"Line {i}") +``` + +## Expander + +```python +import streamlit as st + +# Basic expander +with st.expander("See details"): + st.write("Hidden content here") + +# Start expanded +with st.expander("FAQ", expanded=True): + st.write("Answer to question") +``` + +## Tabs + +```python +import streamlit as st + +tab1, tab2, tab3 = st.tabs(["Data", "Chart", "Settings"]) + +with tab1: + st.dataframe(df) + +with tab2: + st.line_chart(df) + +with tab3: + st.slider("Option", 0, 100) +``` + +## Sidebar + +```python +import streamlit as st + +# Using 'with' syntax +with st.sidebar: + st.title("Controls") + option = st.selectbox("Filter", ["All", "Active", "Archived"]) + +# Using object notation +st.sidebar.title("Navigation") +page = st.sidebar.radio("Go to", ["Home", "Data", "About"]) +``` + +## Empty Placeholders + +```python +import streamlit as st +import time + +# Dynamic content updates +placeholder = st.empty() + +# Replace content +placeholder.text("Loading...") +time.sleep(1) +placeholder.text("Still loading...") +time.sleep(1) +placeholder.success("Done!") + +# Clear content +placeholder.empty() + +# Use as container +with placeholder.container(): + st.write("Multiple elements") + st.write("In a placeholder") +``` + +## Popover + +```python +import streamlit as st + +with st.popover("Settings"): + st.checkbox("Enable notifications") + st.slider("Volume", 0, 100) +``` + +## Dialog (Modal) + +```python +import streamlit as st + +@st.dialog("Confirm Delete") +def confirm_delete(item_name): + st.write(f"Are you sure you want to delete {item_name}?") + col1, col2 = st.columns(2) + if col1.button("Cancel"): + st.rerun() + if col2.button("Delete", type="primary"): + # Perform deletion + st.session_state.deleted = item_name + st.rerun() + +if st.button("Delete Item"): + confirm_delete("My Document") +``` + +## Page Configuration + +```python +import streamlit as st + +# Must be first Streamlit command +st.set_page_config( + page_title="My App", + page_icon="🎯", + layout="wide", # or "centered" + initial_sidebar_state="expanded", # or "collapsed", "auto" + menu_items={ + 'Get Help': 'https://example.com/help', + 'Report a bug': "https://example.com/bug", + 'About': "# My App\nBuilt with Streamlit" + } +) +``` + +## Dashboard Layout Example + +```python +import streamlit as st + +st.set_page_config(layout="wide") + +# Header row +st.title("Dashboard") + +# Metrics row +m1, m2, m3, m4 = st.columns(4) +m1.metric("Users", "1,234", "+12%") +m2.metric("Revenue", "$12.3K", "+8%") +m3.metric("Orders", "456", "-3%") +m4.metric("Rating", "4.8", "+0.2") + +st.divider() + +# Main content with sidebar +with st.sidebar: + date_range = st.date_input("Date Range", []) + category = st.multiselect("Category", ["A", "B", "C"]) + +# Two-column content +chart_col, data_col = st.columns([2, 1]) + +with chart_col: + st.subheader("Trend") + st.line_chart(trend_data) + +with data_col: + st.subheader("Top Items") + st.dataframe(top_items, hide_index=True) + +# Tabbed details +tab1, tab2 = st.tabs(["Details", "Settings"]) +with tab1: + st.write("Detailed view") +with tab2: + st.write("Configuration options") +``` diff --git a/.claude/skills/streamlit/references/state-patterns.md b/.claude/skills/streamlit/references/state-patterns.md new file mode 100644 index 000000000..75787502d --- /dev/null +++ b/.claude/skills/streamlit/references/state-patterns.md @@ -0,0 +1,240 @@ +# Streamlit State Management Patterns + +Best practices for managing state in Streamlit applications. + +## Contents + +- [Session State Basics](#session-state-basics) +- [Widget-State Binding](#widget-state-binding) +- [Callback Pattern](#callback-pattern) +- [Form State Management](#form-state-management) +- [Multi-Step Wizard](#multi-step-wizard) +- [Data Loading with State](#data-loading-with-state) +- [Authentication State](#authentication-state) +- [Cache vs Session State](#cache-vs-session-state) + +## Session State Basics + +```python +import streamlit as st + +# Check and initialize +if "initialized" not in st.session_state: + st.session_state.initialized = True + st.session_state.user = None + st.session_state.data = [] + st.session_state.settings = {"theme": "light"} + +# Access patterns +value = st.session_state.key # Attribute access +value = st.session_state["key"] # Dict access +value = st.session_state.get("key", default) # Safe access + +# Update patterns +st.session_state.key = new_value +st.session_state["key"] = new_value +``` + +## Widget-State Binding + +```python +import streamlit as st + +# Widgets with keys auto-sync to session_state +st.text_input("Name", key="name") +st.number_input("Age", key="age") + +# Values persist across reruns +st.write(f"Name: {st.session_state.name}, Age: {st.session_state.age}") + +# Update widget value programmatically +if st.button("Clear"): + st.session_state.name = "" + st.session_state.age = 0 +``` + +## Callback Pattern + +```python +import streamlit as st + +# Callbacks run BEFORE the main script +def update_total(): + st.session_state.total = ( + st.session_state.quantity * st.session_state.price + ) + +st.number_input("Quantity", key="quantity", on_change=update_total) +st.number_input("Price", key="price", on_change=update_total) + +# Total is always up-to-date +if "total" in st.session_state: + st.metric("Total", f"${st.session_state.total:.2f}") +``` + +## Form State Management + +```python +import streamlit as st + +# Initialize form data +if "form_data" not in st.session_state: + st.session_state.form_data = { + "name": "", + "email": "", + "submitted": False + } + +with st.form("registration"): + name = st.text_input("Name", value=st.session_state.form_data["name"]) + email = st.text_input("Email", value=st.session_state.form_data["email"]) + + if st.form_submit_button("Submit"): + st.session_state.form_data = { + "name": name, + "email": email, + "submitted": True + } + +if st.session_state.form_data["submitted"]: + st.success(f"Registered: {st.session_state.form_data['name']}") +``` + +## Multi-Step Wizard + +```python +import streamlit as st + +# Track wizard state +if "step" not in st.session_state: + st.session_state.step = 1 + st.session_state.wizard_data = {} + +def next_step(): + st.session_state.step += 1 + +def prev_step(): + st.session_state.step -= 1 + +# Step 1 +if st.session_state.step == 1: + st.header("Step 1: Basic Info") + name = st.text_input("Name", key="wizard_name") + if st.button("Next", on_click=next_step): + st.session_state.wizard_data["name"] = name + +# Step 2 +elif st.session_state.step == 2: + st.header("Step 2: Details") + details = st.text_area("Details", key="wizard_details") + col1, col2 = st.columns(2) + col1.button("Back", on_click=prev_step) + if col2.button("Next", on_click=next_step): + st.session_state.wizard_data["details"] = details + +# Step 3 +elif st.session_state.step == 3: + st.header("Step 3: Confirm") + st.write(st.session_state.wizard_data) + st.button("Back", on_click=prev_step) + if st.button("Submit"): + # Process data + st.session_state.step = 1 + st.session_state.wizard_data = {} +``` + +## Data Loading with State + +```python +import streamlit as st +import pandas as pd + +# Preserve loaded data across reruns +if "data" not in st.session_state: + st.session_state.data = None + st.session_state.data_loaded = False + +uploaded = st.file_uploader("Upload CSV") + +if uploaded and not st.session_state.data_loaded: + st.session_state.data = pd.read_csv(uploaded) + st.session_state.data_loaded = True + +if st.session_state.data is not None: + # Data persists even after file uploader clears + st.dataframe(st.session_state.data) + + # Filter without reloading + col = st.selectbox("Filter column", st.session_state.data.columns) + value = st.text_input("Filter value") + + if value: + filtered = st.session_state.data[ + st.session_state.data[col].astype(str).str.contains(value) + ] + st.dataframe(filtered) +``` + +## Authentication State + +```python +import streamlit as st + +# Auth state +if "authenticated" not in st.session_state: + st.session_state.authenticated = False + st.session_state.user = None + +def login(username, password): + # Validate credentials + if username == "admin" and password == "secret": + st.session_state.authenticated = True + st.session_state.user = {"username": username, "role": "admin"} + return True + return False + +def logout(): + st.session_state.authenticated = False + st.session_state.user = None + +# Login form +if not st.session_state.authenticated: + st.title("Login") + with st.form("login"): + username = st.text_input("Username") + password = st.text_input("Password", type="password") + if st.form_submit_button("Login"): + if not login(username, password): + st.error("Invalid credentials") +else: + # Protected content + st.title(f"Welcome, {st.session_state.user['username']}") + st.button("Logout", on_click=logout) +``` + +## Cache vs Session State + +```python +import streamlit as st + +# CACHE: Shared across all users, tied to function inputs +@st.cache_data +def load_global_config(): + """Same result for everyone""" + return load_from_database() + +# SESSION STATE: Per-user, per-session +if "user_preferences" not in st.session_state: + st.session_state.user_preferences = {} + +# Use cache for: +# - Expensive computations +# - Data that doesn't change per user +# - API calls with same parameters + +# Use session state for: +# - User-specific data +# - Form inputs +# - Navigation state +# - Shopping carts, selections +``` diff --git a/.claude/skills/streamlit/references/widgets-catalog.md b/.claude/skills/streamlit/references/widgets-catalog.md new file mode 100644 index 000000000..30f7b5d28 --- /dev/null +++ b/.claude/skills/streamlit/references/widgets-catalog.md @@ -0,0 +1,193 @@ +# Streamlit Widgets Catalog + +Complete reference for all Streamlit input widgets and their usage patterns. + +## Contents + +- [Text Input Widgets](#text-input-widgets) +- [Selection Widgets](#selection-widgets) +- [Numeric Widgets](#numeric-widgets) +- [Date and Time](#date-and-time) +- [Media and Files](#media-and-files) +- [Buttons and Actions](#buttons-and-actions) +- [Widget Keys and Session State](#widget-keys-and-session-state) +- [Widget Callbacks](#widget-callbacks) +- [Disabled and Label Visibility](#disabled-and-label-visibility) +- [Help Text](#help-text) + +## Text Input Widgets + +```python +import streamlit as st + +# Single-line text +name = st.text_input("Name", value="", placeholder="Enter name...") + +# Multi-line text +bio = st.text_area("Bio", height=150) + +# Number input +age = st.number_input("Age", min_value=0, max_value=120, value=25, step=1) + +# Password (masked) +password = st.text_input("Password", type="password") +``` + +## Selection Widgets + +```python +import streamlit as st + +# Dropdown +option = st.selectbox("Choose one", ["A", "B", "C"], index=0) + +# Multi-select +options = st.multiselect("Choose many", ["A", "B", "C"], default=["A"]) + +# Radio buttons +choice = st.radio("Pick one", ["Option 1", "Option 2"], horizontal=True) + +# Checkbox +agree = st.checkbox("I agree", value=False) + +# Toggle +enabled = st.toggle("Enable feature") +``` + +## Numeric Widgets + +```python +import streamlit as st + +# Slider (single value) +value = st.slider("Value", min_value=0, max_value=100, value=50) + +# Range slider +low, high = st.slider("Range", 0, 100, (25, 75)) + +# Select slider (discrete values) +size = st.select_slider("Size", options=["S", "M", "L", "XL"]) +``` + +## Date and Time + +```python +import streamlit as st +from datetime import date, time, datetime + +# Date picker +d = st.date_input("Date", value=date.today()) + +# Date range +start, end = st.date_input("Date range", value=(date(2024, 1, 1), date.today())) + +# Time picker +t = st.time_input("Time", value=time(12, 0)) +``` + +## Media and Files + +```python +import streamlit as st + +# File uploader +file = st.file_uploader("Upload", type=["csv", "xlsx", "pdf"]) +files = st.file_uploader("Upload many", accept_multiple_files=True) + +# Camera input +photo = st.camera_input("Take a photo") + +# Color picker +color = st.color_picker("Pick color", "#FF0000") +``` + +## Buttons and Actions + +```python +import streamlit as st + +# Standard button +if st.button("Click me", type="primary"): + st.write("Clicked!") + +# Download button +st.download_button( + label="Download", + data=csv_data, + file_name="data.csv", + mime="text/csv" +) + +# Link button +st.link_button("Go to docs", "https://docs.streamlit.io") + +# Form submit button (only inside forms) +with st.form("form"): + st.text_input("Name") + st.form_submit_button("Submit") +``` + +## Widget Keys and Session State + +Every widget can have a `key` parameter linking it to session state: + +```python +import streamlit as st + +# Widget with key +st.text_input("Name", key="user_name") + +# Access via session_state +if st.session_state.user_name: + st.write(f"Hello, {st.session_state.user_name}") + +# Programmatically set widget value +if st.button("Reset"): + st.session_state.user_name = "" +``` + +## Widget Callbacks + +Execute code when widgets change: + +```python +import streamlit as st + +def on_name_change(): + # Runs BEFORE the rest of the script + st.session_state.greeting = f"Hello, {st.session_state.name}!" + +st.text_input("Name", key="name", on_change=on_name_change) + +# For buttons +def on_click(): + st.session_state.counter += 1 + +st.button("Increment", on_click=on_click) +``` + +## Disabled and Label Visibility + +```python +import streamlit as st + +# Disable widget +st.text_input("Locked", disabled=True) + +# Hide label (for custom layouts) +st.text_input("Hidden label", label_visibility="hidden") + +# Collapse label +st.text_input("Collapsed", label_visibility="collapsed") +``` + +## Help Text + +```python +import streamlit as st + +st.text_input( + "API Key", + help="Find your API key in the settings page" +) +``` diff --git a/docs/en/enterprise/guides/deploy-crew.mdx b/docs/en/enterprise/guides/deploy-to-amp.mdx similarity index 73% rename from docs/en/enterprise/guides/deploy-crew.mdx rename to docs/en/enterprise/guides/deploy-to-amp.mdx index 8e4c77e41..dc686871c 100644 --- a/docs/en/enterprise/guides/deploy-crew.mdx +++ b/docs/en/enterprise/guides/deploy-to-amp.mdx @@ -1,12 +1,12 @@ --- -title: "Deploy Crew" -description: "Deploying a Crew on CrewAI AMP" +title: "Deploy to AMP" +description: "Deploy your Crew or Flow to CrewAI AMP" icon: "rocket" mode: "wide" --- - After creating a crew locally or through Crew Studio, the next step is + After creating a Crew or Flow locally (or through Crew Studio), the next step is deploying it to the CrewAI AMP platform. This guide covers multiple deployment methods to help you choose the best approach for your workflow. @@ -14,19 +14,26 @@ mode: "wide" ## Prerequisites - - You should have a working crew either built locally or created through Crew - Studio + + You should have a working Crew or Flow that runs successfully locally. + Follow our [preparation guide](/en/enterprise/guides/prepare-for-deployment) to verify your project structure. - Your crew code should be in a GitHub repository (for GitHub integration + Your code should be in a GitHub repository (for GitHub integration method) + + **Crews vs Flows**: Both project types can be deployed as "automations" on CrewAI AMP. + The deployment process is the same, but they have different project structures. + See [Prepare for Deployment](/en/enterprise/guides/prepare-for-deployment) for details. + + ## Option 1: Deploy Using CrewAI CLI -The CLI provides the fastest way to deploy locally developed crews to the Enterprise platform. +The CLI provides the fastest way to deploy locally developed Crews or Flows to the AMP platform. +The CLI automatically detects your project type from `pyproject.toml` and builds accordingly. @@ -128,7 +135,7 @@ crewai deploy remove ## Option 2: Deploy Directly via Web Interface -You can also deploy your crews directly through the CrewAI AMP web interface by connecting your GitHub account. This approach doesn't require using the CLI on your local machine. +You can also deploy your Crews or Flows directly through the CrewAI AMP web interface by connecting your GitHub account. This approach doesn't require using the CLI on your local machine. The platform automatically detects your project type and handles the build appropriately. @@ -387,7 +394,107 @@ The Enterprise platform also offers: - **Custom Tools Repository**: Create, share, and install tools - **Crew Studio**: Build crews through a chat interface without writing code +## Troubleshooting Deployment Failures + +If your deployment fails, check these common issues: + +### Build Failures + +#### Missing uv.lock File + +**Symptom**: Build fails early with dependency resolution errors + +**Solution**: Generate and commit the lock file: + +```bash +uv lock +git add uv.lock +git commit -m "Add uv.lock for deployment" +git push +``` + + + The `uv.lock` file is required for all deployments. Without it, the platform + cannot reliably install your dependencies. + + +#### Wrong Project Structure + +**Symptom**: "Could not find entry point" or "Module not found" errors + +**Solution**: Verify your project matches the expected structure: + +- **Crews**: Must have entry point at `src/project_name/main.py` with a `run()` function +- **Flows**: Must have entry point at `main.py` (project root) with a `kickoff()` function + +See [Prepare for Deployment](/en/enterprise/guides/prepare-for-deployment) for detailed structure diagrams. + +#### Missing CrewBase Decorator + +**Symptom**: "Crew not found", "Config not found", or agent/task configuration errors + +**Solution**: Ensure **all** crew classes use the `@CrewBase` decorator: + +```python +from crewai.project import CrewBase, agent, crew, task + +@CrewBase # This decorator is REQUIRED +class YourCrew(): + """Your crew description""" + + @agent + def my_agent(self) -> Agent: + return Agent( + config=self.agents_config['my_agent'], # type: ignore[index] + verbose=True + ) + + # ... rest of crew definition +``` + + + This applies to standalone Crews AND crews embedded inside Flow projects. + Every crew class needs the decorator. + + +#### Incorrect pyproject.toml Type + +**Symptom**: Build succeeds but runtime fails, or unexpected behavior + +**Solution**: Verify the `[tool.crewai]` section matches your project type: + +```toml +# For Crew projects: +[tool.crewai] +type = "crew" + +# For Flow projects: +[tool.crewai] +type = "flow" +``` + +### Runtime Failures + +#### LLM Connection Failures + +**Symptom**: API key errors, "model not found", or authentication failures + +**Solution**: +1. Verify your LLM provider's API key is correctly set in environment variables +2. Check that variable names don't match [blocked patterns](#blocked-environment-variable-patterns) +3. Test locally with the exact same environment variables before deploying + +#### Crew Execution Errors + +**Symptom**: Crew starts but fails during execution + +**Solution**: +1. Check the execution logs in the AMP dashboard (Traces tab) +2. Verify all tools have required API keys configured +3. Ensure agent configurations in `agents.yaml` are valid +4. Check task configurations in `tasks.yaml` for syntax errors + Contact our support team for assistance with deployment issues or questions - about the Enterprise platform. + about the AMP platform. diff --git a/docs/en/enterprise/guides/prepare-for-deployment.mdx b/docs/en/enterprise/guides/prepare-for-deployment.mdx new file mode 100644 index 000000000..27b30abfe --- /dev/null +++ b/docs/en/enterprise/guides/prepare-for-deployment.mdx @@ -0,0 +1,296 @@ +--- +title: "Prepare for Deployment" +description: "Ensure your Crew or Flow is ready for deployment to CrewAI AMP" +icon: "clipboard-check" +mode: "wide" +--- + + + Before deploying to CrewAI AMP, it's crucial to verify your project is correctly structured. + Both Crews and Flows can be deployed as "automations," but they have different project structures + and requirements that must be met for successful deployment. + + +## Understanding Automations + +In CrewAI AMP, **automations** is the umbrella term for deployable AI projects. An automation can be either: + +- **A Crew**: A standalone team of AI agents working together on tasks +- **A Flow**: An orchestrated workflow that can combine multiple crews, direct LLM calls, and procedural logic + +Understanding which type you're deploying is essential because they have different project structures and entry points. + +## Crews vs Flows: Key Differences + + + + Standalone AI agent teams with a nested `src/` structure. Best for focused, collaborative tasks. + + + Orchestrated workflows with root-level `main.py`. Best for complex, multi-stage processes. + + + +| Aspect | Crew | Flow | +|--------|------|------| +| **Project structure** | Nested `src/project_name/` | Root-level files | +| **Main logic location** | `src/project_name/crew.py` | `main.py` at root | +| **Entry point** | `src/project_name/main.py` | `main.py` at root | +| **pyproject.toml type** | `type = "crew"` | `type = "flow"` | +| **CLI create command** | `crewai create crew name` | `crewai create flow name` | +| **Can contain other crews** | No | Yes (in `crews/` folder) | + +## Project Structure Reference + +### Crew Project Structure + +When you run `crewai create crew my_crew`, you get this structure: + +``` +my_crew/ +├── .gitignore +├── pyproject.toml # Must have type = "crew" +├── README.md +├── .env +├── uv.lock # REQUIRED for deployment +└── src/ + └── my_crew/ + ├── __init__.py + ├── main.py # Entry point with run() function + ├── crew.py # Crew class with @CrewBase decorator + ├── tools/ + │ ├── custom_tool.py + │ └── __init__.py + └── config/ + ├── agents.yaml # Agent definitions + └── tasks.yaml # Task definitions +``` + + + The nested `src/project_name/` structure is critical for Crews. + Placing files at the wrong level will cause deployment failures. + + +### Flow Project Structure + +When you run `crewai create flow my_flow`, you get this structure: + +``` +my_flow/ +├── .gitignore +├── pyproject.toml # Must have type = "flow" +├── README.md +├── .env +├── uv.lock # REQUIRED for deployment +├── main.py # Entry point with kickoff() function +├── crews/ # Embedded crews +│ └── research_crew/ +│ ├── config/ +│ │ ├── agents.yaml +│ │ └── tasks.yaml +│ └── research_crew.py # Each crew needs @CrewBase decorator +└── tools/ + └── custom_tool.py +``` + + + Flows have `main.py` at the **root level**, not inside a `src/` folder. + This is a key difference from Crew projects. + + +## Pre-Deployment Checklist + +Use this checklist to verify your project is ready for deployment. + +### 1. Verify pyproject.toml Configuration + +Your `pyproject.toml` must include the correct `[tool.crewai]` section: + + + + ```toml + [tool.crewai] + type = "crew" + ``` + + + ```toml + [tool.crewai] + type = "flow" + ``` + + + + + If the `type` doesn't match your project structure, the build will fail or + the automation won't run correctly. + + +### 2. Ensure uv.lock File Exists + +CrewAI uses `uv` for dependency management. The `uv.lock` file ensures reproducible builds and is **required** for deployment. + +```bash +# Generate or update the lock file +uv lock + +# Verify it exists +ls -la uv.lock +``` + +If the file doesn't exist, run `uv lock` and commit it to your repository: + +```bash +uv lock +git add uv.lock +git commit -m "Add uv.lock for deployment" +git push +``` + +### 3. Validate CrewBase Decorator Usage + +**Every crew class must use the `@CrewBase` decorator.** This applies to: + +- Standalone crew projects +- Crews embedded inside Flow projects + +```python +from crewai import Agent, Crew, Process, Task +from crewai.project import CrewBase, agent, crew, task +from crewai.agents.agent_builder.base_agent import BaseAgent +from typing import List + +@CrewBase # This decorator is REQUIRED +class MyCrew(): + """My crew description""" + + agents: List[BaseAgent] + tasks: List[Task] + + @agent + def my_agent(self) -> Agent: + return Agent( + config=self.agents_config['my_agent'], # type: ignore[index] + verbose=True + ) + + @task + def my_task(self) -> Task: + return Task( + config=self.tasks_config['my_task'] # type: ignore[index] + ) + + @crew + def crew(self) -> Crew: + return Crew( + agents=self.agents, + tasks=self.tasks, + process=Process.sequential, + verbose=True, + ) +``` + + + If you forget the `@CrewBase` decorator, your deployment will fail with + errors about missing agents or tasks configurations. + + +### 4. Check Project Entry Points + +Verify your project has the correct entry point function: + + + + The entry point is in `src/project_name/main.py`: + + ```python + # src/my_crew/main.py + from my_crew.crew import MyCrew + + def run(): + """Run the crew.""" + inputs = {'topic': 'AI in Healthcare'} + result = MyCrew().crew().kickoff(inputs=inputs) + return result + + if __name__ == "__main__": + run() + ``` + + + The entry point is in `main.py` at the project root: + + ```python + # main.py (at project root) + from crewai.flow.flow import Flow, listen, start + + class MyFlow(Flow): + @start() + def begin(self): + # Flow logic here + pass + + def kickoff(): + """Run the flow.""" + MyFlow().kickoff() + + if __name__ == "__main__": + kickoff() + ``` + + + +### 5. Prepare Environment Variables + +Before deployment, ensure you have: + +1. **LLM API keys** ready (OpenAI, Anthropic, Google, etc.) +2. **Tool API keys** if using external tools (Serper, etc.) +3. Variable names that comply with [security requirements](/en/enterprise/guides/deploy-to-amp#environment-variable-security-requirements) + + + Test your project locally with the same environment variables before deploying + to catch configuration issues early. + + +## Quick Validation Commands + +Run these commands from your project root to quickly verify your setup: + +```bash +# 1. Check project type in pyproject.toml +grep -A2 "\[tool.crewai\]" pyproject.toml + +# 2. Verify uv.lock exists +ls -la uv.lock || echo "ERROR: uv.lock missing! Run 'uv lock'" + +# 3. For Crews - verify structure +ls -la src/*/crew.py 2>/dev/null || echo "No crew.py found in src/" + +# 4. For Flows - verify structure +ls -la main.py 2>/dev/null || echo "No main.py at root" +ls -la crews/ 2>/dev/null || echo "No crews/ folder (optional for flows)" + +# 5. Check for CrewBase usage +grep -r "@CrewBase" . --include="*.py" +``` + +## Common Setup Mistakes + +| Mistake | Symptom | Fix | +|---------|---------|-----| +| Missing `uv.lock` | Build fails during dependency resolution | Run `uv lock` and commit | +| Wrong `type` in pyproject.toml | Build succeeds but runtime fails | Change to correct type | +| Missing `@CrewBase` decorator | "Config not found" errors | Add decorator to all crew classes | +| Crew `main.py` at root level | Entry point not found | Move to `src/project_name/` | +| Flow `main.py` in `src/` | Entry point not found | Move to project root | +| Missing `run()` or `kickoff()` | Cannot start automation | Add correct entry function | + +## Next Steps + +Once your project passes all checklist items, you're ready to deploy: + + + Follow the deployment guide to deploy your Crew or Flow to CrewAI AMP using + the CLI, web interface, or CI/CD integration. +