Brandon/cre 130 pipeline project structure (#1066)

* WIP. Procedure appears to be working well. Working on mocking properly for tests

* All tests are passing now

* rshift working

* Add back in Gui's tool_usage fix

* WIP

* Going to start refactoring for pipeline_output

* Update terminology

* new pipeline flow with traces and usage metrics working. need to add more tests and make sure PipelineOutput behaves likew CrewOutput

* Fix pipelineoutput to look more like crewoutput and taskoutput

* Implemented additional tests for pipeline. One test is failing. Need team support

* Update docs for pipeline

* Update pipeline to properly process input and ouput dictionary

* Update Pipeline docs

* Add back in commentary at top of pipeline file

* Starting to work on router

* Drop router for now. will add in separately

* In the middle of fixing router. A ton of circular dependencies. Moving over to a new design.

* WIP.

* Fix circular dependencies and updated PipelineRouter

* Add in Eduardo feedback. Still need to add in more commentary describing the design decisions for pipeline

* Add developer notes to explain what is going on in pipelines.

* Add doc strings

* Fix missing rag datatype

* WIP. Converting usage metrics from a dict to an object

* Fix tests that were checking usage metrics

* Drop todo

* Fix 1 type error in pipeline

* Update pipeline to use UsageMetric

* Add missing doc string

* WIP.

* Change names

* Rename variables based on joaos feedback

* Fix critical circular dependency issues. Now needing to fix trace issue.

* Tests working now!

* Add more tests which showed underlying issue with traces

* Fix tests

* Remove overly complicated test

* Add router example to docs

* Clean up end of docs

* Clean up docs

* Working on creating Crew templates and pipeline templates

* WIP.

* WIP

* Fix poetry install from templates

* WIP

* Restructure

* changes for lorenze

* more todos

* WIP: create pipelines cli working

* wrapped up router

* ignore mypy src on templates

* ignored signature of copy

* fix all verbose

* rm print statements

* brought back correct folders

* fixes missing folders and then rm print statements

* fixed tests

* fixed broken test

* fixed type checker

* fixed type ignore

* ignore types for templates

* needed

* revert

* exclude only required

* rm type errors on templates

* rm excluding type checks for template files on github action

* fixed missing quotes

---------

Co-authored-by: Brandon Hancock <brandon@brandonhancock.io>

src/crewai/cli/templates/pipeline_router/.gitignore

@@ -0,0 +1,2 @@
+.env
+__pycache__/

src/crewai/cli/templates/pipeline_router/README.md

@@ -0,0 +1,57 @@
+# {{crew_name}} Crew
+
+Welcome to the {{crew_name}} Crew project, powered by [crewAI](https://crewai.com). This template is designed to help you set up a multi-agent AI system with ease, leveraging the powerful and flexible framework provided by crewAI. Our goal is to enable your agents to collaborate effectively on complex tasks, maximizing their collective intelligence and capabilities.
+
+## Installation
+
+Ensure you have Python >=3.10 <=3.13 installed on your system. This project uses [Poetry](https://python-poetry.org/) for dependency management and package handling, offering a seamless setup and execution experience.
+
+First, if you haven't already, install Poetry:
+
+```bash
+pip install poetry
+```
+
+Next, navigate to your project directory and install the dependencies:
+
+1. First lock the dependencies and then install them:
+```bash
+poetry lock
+```
+```bash
+poetry install
+```
+### Customizing
+
+**Add your `OPENAI_API_KEY` into the `.env` file**
+
+- Modify `src/{{folder_name}}/config/agents.yaml` to define your agents
+- Modify `src/{{folder_name}}/config/tasks.yaml` to define your tasks
+- Modify `src/{{folder_name}}/crew.py` to add your own logic, tools and specific args
+- Modify `src/{{folder_name}}/main.py` to add custom inputs for your agents and tasks
+
+## Running the Project
+
+To kickstart your crew of AI agents and begin task execution, run this from the root folder of your project:
+
+```bash
+poetry run {{folder_name}}
+```
+
+This command initializes the {{name}} Crew, assembling the agents and assigning them tasks as defined in your configuration.
+
+This example, unmodified, will run the create a `report.md` file with the output of a research on LLMs in the root folder.
+
+## Understanding Your Crew
+
+The {{name}} Crew is composed of multiple AI agents, each with unique roles, goals, and tools. These agents collaborate on a series of tasks, defined in `config/tasks.yaml`, leveraging their collective skills to achieve complex objectives. The `config/agents.yaml` file outlines the capabilities and configurations of each agent in your crew.
+
+## Support
+
+For support, questions, or feedback regarding the {{crew_name}} Crew or crewAI.
+- Visit our [documentation](https://docs.crewai.com)
+- Reach out to us through our [GitHub repository](https://github.com/joaomdmoura/crewai)
+- [Join our Discord](https://discord.com/invite/X4JWnZnxPb)
+- [Chat with our docs](https://chatg.pt/DWjSBZn)
+
+Let's create wonders together with the power and simplicity of crewAI.

src/crewai/cli/templates/pipeline_router/config/agents.yaml

@@ -0,0 +1,19 @@
+researcher:
+  role: >
+    {topic} Senior Data Researcher
+  goal: >
+    Uncover cutting-edge developments in {topic}
+  backstory: >
+    You're a seasoned researcher with a knack for uncovering the latest
+    developments in {topic}. Known for your ability to find the most relevant
+    information and present it in a clear and concise manner.
+
+reporting_analyst:
+  role: >
+    {topic} Reporting Analyst
+  goal: >
+    Create detailed reports based on {topic} data analysis and research findings
+  backstory: >
+    You're a meticulous analyst with a keen eye for detail. You're known for
+    your ability to turn complex data into clear and concise reports, making
+    it easy for others to understand and act on the information you provide.

src/crewai/cli/templates/pipeline_router/config/tasks.yaml

@@ -0,0 +1,17 @@
+research_task:
+  description: >
+    Conduct a thorough research about {topic}
+    Make sure you find any interesting and relevant information given
+    the current year is 2024.
+  expected_output: >
+    A list with 10 bullet points of the most relevant information about {topic}
+  agent: researcher
+
+reporting_task:
+  description: >
+    Review the context you got and expand each topic into a full section for a report.
+    Make sure the report is detailed and contains any and all relevant information.
+  expected_output: >
+    A fully fledge reports with the mains topics, each with a full section of information.
+    Formatted as markdown without '```'
+  agent: reporting_analyst

src/crewai/cli/templates/pipeline_router/crews/classifier_crew/classifier_crew.py

@@ -0,0 +1,40 @@
+from crewai import Agent, Crew, Process, Task
+from crewai.project import CrewBase, agent, crew, task
+from pydantic import BaseModel
+
+# Uncomment the following line to use an example of a custom tool
+# from demo_pipeline.tools.custom_tool import MyCustomTool
+
+# Check our tools documentations for more information on how to use them
+# from crewai_tools import SerperDevTool
+
+class UrgencyScore(BaseModel):
+    urgency_score: int
+
+@CrewBase
+class ClassifierCrew:
+    """Email Classifier Crew"""
+
+    agents_config = "config/agents.yaml"
+    tasks_config = "config/tasks.yaml"
+
+    @agent
+    def classifier(self) -> Agent:
+        return Agent(config=self.agents_config["classifier"], verbose=True)
+
+    @task
+    def urgent_task(self) -> Task:
+        return Task(
+            config=self.tasks_config["classify_email"],
+            output_pydantic=UrgencyScore,
+        )
+
+    @crew
+    def crew(self) -> Crew:
+        """Creates the Email Classifier Crew"""
+        return Crew(
+            agents=self.agents,  # Automatically created by the @agent decorator
+            tasks=self.tasks,  # Automatically created by the @task decorator
+            process=Process.sequential,
+            verbose=True,
+        )

src/crewai/cli/templates/pipeline_router/crews/classifier_crew/config/agents.yaml

@@ -0,0 +1,7 @@
+classifier:
+  role: >
+    Email Classifier
+  goal: >
+    Classify the email: {email} as urgent or normal from a score of 1 to 10, where 1 is not urgent and 10 is urgent. Return the urgency score only.`
+  backstory: >
+    You are a highly efficient and experienced email classifier, trained to quickly assess and classify emails. Your ability to remain calm under pressure and provide concise, actionable responses has made you an invaluable asset in managing normal situations and maintaining smooth operations.

src/crewai/cli/templates/pipeline_router/crews/classifier_crew/config/tasks.yaml

@@ -0,0 +1,7 @@
+classify_email:
+  description: >
+    Classify the email: {email}
+    as urgent or normal.
+  expected_output: >
+    Classify the email from a scale of 1 to 10, where 1 is not urgent and 10 is urgent. Return the urgency score only.
+  agent: classifier

src/crewai/cli/templates/pipeline_router/crews/normal_crew/config/agents.yaml

@@ -0,0 +1,7 @@
+normal_handler:
+  role: >
+    Normal Email Processor
+  goal: >
+    Process normal emails and create an email to respond to the sender.
+  backstory: >
+    You are a highly efficient and experienced normal email handler, trained to quickly assess and respond to normal communications. Your ability to remain calm under pressure and provide concise, actionable responses has made you an invaluable asset in managing normal situations and maintaining smooth operations.

src/crewai/cli/templates/pipeline_router/crews/normal_crew/config/tasks.yaml

@@ -0,0 +1,6 @@
+normal_task:
+  description: >
+    Process and respond to normal email quickly.
+  expected_output: >
+    An email response to the normal email.
+  agent: normal_handler

src/crewai/cli/templates/pipeline_router/crews/normal_crew/normal_crew.py

@@ -0,0 +1,36 @@
+from crewai import Agent, Crew, Process, Task
+from crewai.project import CrewBase, agent, crew, task
+
+# Uncomment the following line to use an example of a custom tool
+# from demo_pipeline.tools.custom_tool import MyCustomTool
+
+# Check our tools documentations for more information on how to use them
+# from crewai_tools import SerperDevTool
+
+
+@CrewBase
+class NormalCrew:
+    """Normal Email Crew"""
+
+    agents_config = "config/agents.yaml"
+    tasks_config = "config/tasks.yaml"
+
+    @agent
+    def normal_handler(self) -> Agent:
+        return Agent(config=self.agents_config["normal_handler"], verbose=True)
+
+    @task
+    def urgent_task(self) -> Task:
+        return Task(
+            config=self.tasks_config["normal_task"],
+        )
+
+    @crew
+    def crew(self) -> Crew:
+        """Creates the Normal Email Crew"""
+        return Crew(
+            agents=self.agents,  # Automatically created by the @agent decorator
+            tasks=self.tasks,  # Automatically created by the @task decorator
+            process=Process.sequential,
+            verbose=True,
+        )

src/crewai/cli/templates/pipeline_router/crews/urgent_crew/config/agents.yaml

@@ -0,0 +1,7 @@
+urgent_handler:
+  role: >
+    Urgent Email Processor
+  goal: >
+    Process urgent emails and create an email to respond to the sender.
+  backstory: >
+    You are a highly efficient and experienced urgent email handler, trained to quickly assess and respond to time-sensitive communications. Your ability to remain calm under pressure and provide concise, actionable responses has made you an invaluable asset in managing critical situations and maintaining smooth operations.

src/crewai/cli/templates/pipeline_router/crews/urgent_crew/config/tasks.yaml

@@ -0,0 +1,6 @@
+urgent_task:
+  description: >
+    Process and respond to urgent email quickly.
+  expected_output: >
+    An email response to the urgent email.
+  agent: urgent_handler

src/crewai/cli/templates/pipeline_router/crews/urgent_crew/urgent_crew.py

@@ -0,0 +1,36 @@
+from crewai import Agent, Crew, Process, Task
+from crewai.project import CrewBase, agent, crew, task
+
+# Uncomment the following line to use an example of a custom tool
+# from demo_pipeline.tools.custom_tool import MyCustomTool
+
+# Check our tools documentations for more information on how to use them
+# from crewai_tools import SerperDevTool
+
+
+@CrewBase
+class UrgentCrew:
+    """Urgent Email Crew"""
+
+    agents_config = "config/agents.yaml"
+    tasks_config = "config/tasks.yaml"
+
+    @agent
+    def urgent_handler(self) -> Agent:
+        return Agent(config=self.agents_config["urgent_handler"], verbose=True)
+
+    @task
+    def urgent_task(self) -> Task:
+        return Task(
+            config=self.tasks_config["urgent_task"],
+        )
+
+    @crew
+    def crew(self) -> Crew:
+        """Creates the Urgent Email Crew"""
+        return Crew(
+            agents=self.agents,  # Automatically created by the @agent decorator
+            tasks=self.tasks,  # Automatically created by the @task decorator
+            process=Process.sequential,
+            verbose=True,
+        )

src/crewai/cli/templates/pipeline_router/main.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+import asyncio
+from crewai.routers.router import Route
+from crewai.routers.router import Router
+
+from {{folder_name}}.pipelines.pipeline_classifier import EmailClassifierPipeline
+from {{folder_name}}.pipelines.pipeline_normal import NormalPipeline
+from {{folder_name}}.pipelines.pipeline_urgent import UrgentPipeline
+
+async def run():
+    """
+    Run the pipeline.
+    """
+    inputs = [
+       {
+        "email": """
+            Subject: URGENT: Marketing Campaign Launch - Immediate Action Required
+            Dear Team,
+            I'm reaching out regarding our upcoming marketing campaign that requires your immediate attention and swift action. We're facing a critical deadline, and our success hinges on our ability to mobilize quickly.
+            Key points:
+            
+            Campaign launch: 48 hours from now
+            Target audience: 250,000 potential customers
+            Expected ROI: 35% increase in Q3 sales
+            
+            What we need from you NOW:
+            
+            Final approval on creative assets (due in 3 hours)
+            Confirmation of media placements (due by end of day)
+            Last-minute budget allocation for paid social media push
+            
+            Our competitors are poised to launch similar campaigns, and we must act fast to maintain our market advantage. Delays could result in significant lost opportunities and potential revenue.
+            Please prioritize this campaign above all other tasks. I'll be available for the next 24 hours to address any concerns or roadblocks.
+            Let's make this happen!
+            [Your Name]
+            Marketing Director
+            P.S. I'll be scheduling an emergency team meeting in 1 hour to discuss our action plan. Attendance is mandatory.
+        """
+       }
+    ]
+
+    pipeline_classifier = EmailClassifierPipeline().create_pipeline()
+    pipeline_urgent = UrgentPipeline().create_pipeline()
+    pipeline_normal = NormalPipeline().create_pipeline()
+
+    router = Router(
+        routes={
+            "high_urgency": Route(
+                condition=lambda x: x.get("urgency_score", 0) > 7,
+                pipeline=pipeline_urgent
+            ),
+            "low_urgency": Route(
+                condition=lambda x: x.get("urgency_score", 0) <= 7,
+                pipeline=pipeline_normal
+            )
+        },
+        default=pipeline_normal
+    )
+
+    pipeline = pipeline_classifier >> router
+
+    results = await pipeline.kickoff(inputs)
+
+    # Process and print results
+    for result in results:
+        print(f"Raw output: {result.raw}")
+        if result.json_dict:
+            print(f"JSON output: {result.json_dict}")
+        print("\n")
+
+def main():
+    asyncio.run(run())
+
+if __name__ == "__main__":
+    main()

src/crewai/cli/templates/pipeline_router/pipelines/pipeline_classifier.py

@@ -0,0 +1,24 @@
+from crewai import Pipeline
+from crewai.project import PipelineBase
+from ..crews.classifier_crew.classifier_crew import ClassifierCrew
+
+
+@PipelineBase
+class EmailClassifierPipeline:
+    def __init__(self):
+        # Initialize crews
+        self.classifier_crew = ClassifierCrew().crew()
+
+    def create_pipeline(self):
+        return Pipeline(
+            stages=[
+                self.classifier_crew
+            ]
+        )
+    
+    async def kickoff(self, inputs):
+        pipeline = self.create_pipeline()
+        results = await pipeline.kickoff(inputs)
+        return results
+
+

src/crewai/cli/templates/pipeline_router/pipelines/pipeline_normal.py

@@ -0,0 +1,24 @@
+from crewai import Pipeline
+from crewai.project import PipelineBase
+from ..crews.normal_crew.normal_crew import NormalCrew
+
+
+@PipelineBase
+class NormalPipeline:
+    def __init__(self):
+        # Initialize crews
+        self.normal_crew = NormalCrew().crew()
+
+    def create_pipeline(self):
+        return Pipeline(
+            stages=[
+                self.normal_crew
+            ]
+        )
+    
+    async def kickoff(self, inputs):
+        pipeline = self.create_pipeline()
+        results = await pipeline.kickoff(inputs)
+        return results
+
+

src/crewai/cli/templates/pipeline_router/pipelines/pipeline_urgent.py

@@ -0,0 +1,23 @@
+from crewai import Pipeline
+from crewai.project import PipelineBase
+from ..crews.urgent_crew.urgent_crew import UrgentCrew
+
+@PipelineBase
+class UrgentPipeline:
+    def __init__(self):
+        # Initialize crews
+        self.urgent_crew = UrgentCrew().crew()
+    
+    def create_pipeline(self):
+        return Pipeline(
+            stages=[
+                self.urgent_crew
+            ]
+        )
+    
+    async def kickoff(self, inputs):
+        pipeline = self.create_pipeline()
+        results = await pipeline.kickoff(inputs)
+        return results
+
+

src/crewai/cli/templates/pipeline_router/pyproject.toml

@@ -0,0 +1,19 @@
+[tool.poetry]
+name = "{{folder_name}}"
+version = "0.1.0"
+description = "{{name}} using crewAI"
+authors = ["Your Name <you@example.com>"]
+
+[tool.poetry.dependencies]
+python = ">=3.10,<=3.13"
+crewai = { extras = ["tools"], version = "^0.46.0" }
+
+[tool.poetry.scripts]
+{{folder_name}} = "{{folder_name}}.main:main"
+train = "{{folder_name}}.main:train"
+replay = "{{folder_name}}.main:replay"
+test = "{{folder_name}}.main:test"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

src/crewai/cli/templates/pipeline_router/tools/custom_tool.py

@@ -0,0 +1,12 @@
+from crewai_tools import BaseTool
+
+
+class MyCustomTool(BaseTool):
+    name: str = "Name of my tool"
+    description: str = (
+        "Clear description for what this tool is useful for, you agent will need this information to use it."
+    )
+
+    def _run(self, argument: str) -> str:
+        # Implementation goes here
+        return "this is an example of a tool output, ignore it and move along."