Commit Graph

51 Commits

Author SHA1 Message Date
Lorenze Jay
4c122321ad Brandon/cre 130 pipeline project structure (#1066)
* WIP. Procedure appears to be working well. Working on mocking properly for tests

* All tests are passing now

* rshift working

* Add back in Gui's tool_usage fix

* WIP

* Going to start refactoring for pipeline_output

* Update terminology

* new pipeline flow with traces and usage metrics working. need to add more tests and make sure PipelineOutput behaves likew CrewOutput

* Fix pipelineoutput to look more like crewoutput and taskoutput

* Implemented additional tests for pipeline. One test is failing. Need team support

* Update docs for pipeline

* Update pipeline to properly process input and ouput dictionary

* Update Pipeline docs

* Add back in commentary at top of pipeline file

* Starting to work on router

* Drop router for now. will add in separately

* In the middle of fixing router. A ton of circular dependencies. Moving over to a new design.

* WIP.

* Fix circular dependencies and updated PipelineRouter

* Add in Eduardo feedback. Still need to add in more commentary describing the design decisions for pipeline

* Add developer notes to explain what is going on in pipelines.

* Add doc strings

* Fix missing rag datatype

* WIP. Converting usage metrics from a dict to an object

* Fix tests that were checking usage metrics

* Drop todo

* Fix 1 type error in pipeline

* Update pipeline to use UsageMetric

* Add missing doc string

* WIP.

* Change names

* Rename variables based on joaos feedback

* Fix critical circular dependency issues. Now needing to fix trace issue.

* Tests working now!

* Add more tests which showed underlying issue with traces

* Fix tests

* Remove overly complicated test

* Add router example to docs

* Clean up end of docs

* Clean up docs

* Working on creating Crew templates and pipeline templates

* WIP.

* WIP

* Fix poetry install from templates

* WIP

* Restructure

* changes for lorenze

* more todos

* WIP: create pipelines cli working

* wrapped up router

* ignore mypy src on templates

* ignored signature of copy

* fix all verbose

* rm print statements

* brought back correct folders

* fixes missing folders and then rm print statements

* fixed tests

* fixed broken test

* fixed type checker

* fixed type ignore

* ignore types for templates

* needed

* revert

* exclude only required

* rm type errors on templates

* rm excluding type checks for template files on github action

* fixed missing quotes

---------

Co-authored-by: Brandon Hancock <brandon@brandonhancock.io>
2024-08-09 14:13:29 -07:00
Lorenze Jay
8eea22c763 Fix logging types to bool (#1051)
* fixes pydantic validations hierarchical

* more tests

* logger logs everything or not

* verbose rm levels to bool

* updated readme verbose levels
2024-08-07 10:31:18 -07:00
Lorenze Jay
071122791e Feat/sliding context window (#1042)
* patching for non-gpt model

* removal of json_object tool name assignment

* fixed issue for smaller models due to instructions prompt

* fixing for ollama llama3 models

* WIP: generated summary from documents split, could also create memgpt approach

* WIP: need tests but user inputted summarization strategy implemented - handling context window exceeding errors

* rm extra line

* removed type ignores

* added tests

* handling n to summarize prompt

* code cleanup, using click for cli asker

* rm not used class

* better refactor

* reverted poetry lock

* reverted poetry.locl

* improved context window exceeding exception class
2024-08-01 13:15:50 -07:00
Lorenze Jay
38f9c3c500 WIP fixed mypy src types (#1036) 2024-07-30 10:59:50 -07:00
Taleb
ca9deaebb7 Performed spell check across the rest of code base, and enahnced the yaml paraser code a little (#895)
* Performed spell check across the entire documentation

Thank you once again!

* Performed spell check across the most of code base
Folders been checked:
- agents
- cli
- memory
- project
- tasks
- telemetry
- tools
- translations

* Trying to add a max_token for the agents, so they limited by number of tokens.

* Performed spell check across the rest of code base, and enahnced the yaml paraser code a little

* Small change in the main agent doc

* Improve _save_file method to handle both dict and str inputs

- Add check for dict type input
- Use json.dump for dict serialization
- Convert non-dict inputs to string
- Remove type ignore comments

---------

Co-authored-by: João Moura <joaomdmoura@gmail.com>
2024-07-28 15:39:54 -03:00
João Moura
a1b37f073d Adding better support for open source tool calling models (#952)
* Adding better support for open source tool calling models

* making sure the right tool is called

* fixing tests

* better support opensource models
2024-07-17 05:54:13 -03:00
Gui Vieira
02f6da700a Introduce structure keys (#902)
* Introduce structure keys

* Add agent key to tasks

* Rebasing is hard

* Rename task output telemetry

* Feedback
2024-07-15 19:37:07 -03:00
Lorenze Jay
387528c3fb Replay feat using db (#930)
* Cleaned up task execution to now have separate paths for async and sync execution. Updating all kickoff functions to return CrewOutput. WIP. Waiting for Joao feedback on async task execution with task_output

* Consistently storing async and sync output for context

* outline tests I need to create going forward

* Major rehaul of TaskOutput and CrewOutput. Updated all tests to work with new change. Need to add in a few final tricky async tests and add a few more to verify output types on TaskOutput and CrewOutput.

* Encountering issues with callback. Need to test on main. WIP

* working on tests. WIP

* WIP. Figuring out disconnect issue.

* Cleaned up logs now that I've isolated the issue to the LLM

* more wip.

* WIP. It looks like usage metrics has always been broken for async

* Update parent crew who is managing for_each loop

* Merge in main to bugfix/kickoff-for-each-usage-metrics

* Clean up code for review

* Add new tests

* Final cleanup. Ready for review.

* Moving copy functionality from Agent to BaseAgent

* Fix renaming issue

* Fix linting errors

* use BaseAgent instead of Agent where applicable

* Fixing missing function. Working on tests.

* WIP. Needing team to review change

* Fixing issues brought about by merge

* WIP: need to fix json encoder

* WIP need to fix encoder

* WIP

* WIP: replay working with async. need to add tests

* Implement major fixes from yesterdays group conversation. Now working on tests.

* The majority of tasks are working now. Need to fix converter class

* Fix final failing test

* Fix linting and type-checker issues

* Add more tests to fully test CrewOutput and TaskOutput changes

* Add in validation for async cannot depend on other async tasks.

* WIP: working replay feat fixing inputs, need tests

* WIP: core logic of seq and heir for executing tasks added into one

* Update validators and tests

* better logic for seq and hier

* replay working for both seq and hier just need tests

* fixed context

* added cli command + code cleanup TODO: need better refactoring

* refactoring for cleaner code

* added better tests

* removed todo comments and fixed some tests

* fix logging now all tests should pass

* cleaner code

* ensure replay is delcared when replaying specific tasks

* ensure hierarchical works

* better typing for stored_outputs and separated task_output_handler

* added better tests

* added replay feature to crew docs

* easier cli command name

* fixing changes

* using sqllite instead of .json file for logging previous task_outputs

* tools fix

* added to docs and fixed tests

* fixed .db

* fixed docs and removed unneeded comments

* separating ltm and replay db

* fixed printing colors

* added how to doc

---------

Co-authored-by: Brandon Hancock <brandon@brandonhancock.io>
2024-07-15 17:14:10 -03:00
Brandon Hancock (bhancock_ai)
93d6655228 Feature/use converter instead of manually trimming (#894)
* Exploring output being passed to tool selector to see if we can better format data

* WIP. Adding JSON repair functionality

* Almost done implementing JSON repair. Testing fixes vs current base case.

* More action cleanup with additional tests

* WIP. Trying to figure out what is going on with tool descriptions

* Update tool description generation

* WIP. Trying to find out what is causing the tools to duplicate

* Replacing tools properly instead of duplicating them accidentally

* Fixing issues for MR

* Update dependencies for JSON_REPAIR

* More cleaning up pull request

* preppering for call

* Fix type-checking issues

---------

Co-authored-by: João Moura <joaomdmoura@gmail.com>
2024-07-15 08:53:41 -03:00
Taleb
ea5811735a Performed spell check across most of code base (#882)
* Performed spell check across the entire documentation

Thank you once again!

* Performed spell check across the most of code base
Folders been checked:
- agents
- cli
- memory
- project
- tasks
- telemetry
- tools
- translations
2024-07-07 13:00:05 -03:00
João Moura
b65eee45a8 TYPO 2024-07-06 00:34:50 -04:00
Eduardo Chiarotti
8dfc40ff98 fix: Fix tests (#873)
* fix: call asserts

* fix: test_increment_tool_errors

* fix: test_increment_delegations_for_sequential_process

* fix: test_increment_delegations_for_hierarchical_process

* fix: test_code_execution_flag_adds_code_tool_upon_kickoff

* fix: test_tool_usage_information_is_appended_to_agent

* fix: try to fix test_crew_full_output

* fix: try to fix test_crew_full_output

* fix: test remove vcr to test crew_test test

* fix: comment test to see if ci passes

* fix: comment test to see if ci passes

* fix: test changing prompt tokens to get error on CI

* fix: test changing prompt tokens to get error on CI

* fix: test changing prompt tokens to get error on CI

* fix: test changing prompt tokens to get error on CI

* fix: test new approach

* fix: comment funciont not working in CI

* fix: github python version

* fix: remove need of vcr

* fix: fix and add comments for all type checking errors
2024-07-05 09:06:56 -03:00
João Moura
b51cf24f0f Adding support to force a tool return to be the final answer. (#867)
* Adding support to force a tool return to be the final answer.
This will at the end of the execution return the tool output.
It will return the output of the latest tool with the flag

* Update src/crewai/agent.py

Co-authored-by: Gui Vieira <guilherme_vieira@me.com>

* Update tests/agent_test.py

Co-authored-by: Gui Vieira <guilherme_vieira@me.com>

---------

Co-authored-by: Gui Vieira <guilherme_vieira@me.com>
2024-07-04 16:36:00 -03:00
João Moura
62e662a690 TYPO 2024-07-03 18:41:52 -04:00
Brandon Hancock (bhancock_ai)
0b277e066d Bugfix/kickoff for each usage metrics (#844)
* WIP. Figuring out disconnect issue.

* Cleaned up logs now that I've isolated the issue to the LLM

* more wip.

* WIP. It looks like usage metrics has always been broken for async

* Update parent crew who is managing for_each loop

* Merge in main to bugfix/kickoff-for-each-usage-metrics

* Clean up code for review

* Add new tests

* Final cleanup. Ready for review.

* Moving copy functionality from Agent to BaseAgent

* Fix renaming issue

* Fix linting errors

* use BaseAgent instead of Agent where applicable
2024-07-03 15:30:53 -03:00
Lorenze Jay
5f0af16616 fixed mixin (#831)
* fixed mixin

* WIP: fixing types

* type fixes on mixin
2024-07-02 21:16:26 -03:00
João Moura
dee7bd6258 small refractoring for new version 2024-07-01 05:32:22 -07:00
Lorenze Jay
9678d22724 Lorenzejay/byoa (#776)
* better spacing

* works with llama index

* works on langchain custom just need delegation to work

* cleanup for custom_agent class

* works with different argument expectations for agent_executor

* cleanup for hierarchial process, better agent_executor args handler and added to the crew agent doc page

* removed code examples for langchain + llama index, added to docs instead

* added key output if return is not a str for and added some tests

* added hinting for CustomAgent class

* removed pass as it was not needed

* closer just need to figuire ou agentTools

* running agents - llamaindex and langchain with base agent

* some cleanup on baseAgent

* minimum for agent to run for base class and ensure it works with hierarchical process

* cleanup for original agent to take on BaseAgent class

* Agent takes on langchainagent and cleanup across

* token handling working for usage_metrics to continue working

* installed llama-index, updated docs and added better name

* fixed some type errors

* base agent holds token_process

* heirarchail process uses proper tools and no longer relies on hasattr for token_processes

* removal of test_custom_agent_executions

* this fixes copying agents

* leveraging an executor class for trigger llamaindex agent

* llama index now has ask_human

* executor mixins added

* added output converter base class

* type listed

* cleanup for output conversions and tokenprocess eliminated redundancy

* properly handling tokens

* simplified token calc handling

* original agent with base agent builder structure setup

* better docs

* no more llama-index dep

* cleaner docs

* test fixes

* poetry reverts and better docs

* base_agent_tools set for third party agents

* updated task and test fix
2024-06-27 14:56:08 -03:00
Eduardo Chiarotti
4e1506f68e fix: fix trainig_data error (#820)
* fix: fix trainig_data error

* fix: fix lack crew on agent

* fix: fix lack crew on agent executor
2024-06-27 12:58:20 -03:00
Eduardo Chiarotti
3573a61568 feat: Add Train feature for Crews (#686)
* feat: add training logic to agent and crew

* feat: add training logic to agent executor

* feat: add input parameter  to cli command

* feat: add utilities for the training logic

* feat: polish code, logic and add private variables

* feat: add docstring and type hinting to executor

* feat: add constant file, add constant to code

* feat: fix name of training handler function

* feat: remove unused var

* feat: change file handler file name

* feat: Add training handler file, class and change on the code

* feat: fix name error from file

* fix: change import to adapt to logic

* feat: add training handler test

* feat: add tests for file and training_handler

* feat: add test for task evaluator function

* feat: change text to fit in-screen

* feat: add test for train function

* feat: add test for agent training_handler function

* feat: add test for agent._use_trained_data
2024-06-27 02:22:34 -03:00
Brandon Hancock (bhancock_ai)
72e20a5dbb Resolved Merge Conflicts for PR #712: Remove Hyphen in co-workers (#786)
* removed hyphen in co-workers

* Fix issue with AgentTool agent selection. The LLM included double quotes in the agent name which messed up the string comparison. Added additional types. Cleaned up error messaging.

* Remove duplicate import

* Improve explanation

* Revert poetry.lock changes

* Fix missing line in poetry.lock

---------

Co-authored-by: madmag77 <goncharov.artemv@gmail.com>
2024-06-18 16:57:56 -03:00
Eduardo Chiarotti
6eb0c0666b feat: add mypy as type checker, update code and add comment to reference (#591)
* fix: fix test actually running

* fix: fix test to not send request to openai

* fix: fix linting to remove cli files

* fix: exclude only files that breaks black

* fix: Fix all Ruff checkings on the code and Fix Test with repeated name

* fix: Change linter name on yml file

* feat: update pre-commit

* feat: remove need for isort on the code

* feat: add mypy as type checker, update code and add comment to reference

* feat: remove black linter

* feat: remove poetry to run the command

* feat: change logic to test mypy

* feat: update tests yml to try to fix the tests gh action

* feat: try to add just mypy to run on gh action

* feat: fix yml file

* feat: add comment to avoid issue on gh action

* feat: decouple pytest from the necessity of poetry install

* feat: change tests.yml to test different approach

* feat: change to poetry run

* fix: parameter field on yml file

* fix: update parameters to be on the pyproject

* fix: update pyproject to remove import untyped errors
2024-05-10 16:37:52 -03:00
João Moura
84775373a6 Adding support for system, prompt and answe templates 2024-05-02 02:21:59 -03:00
João Moura
6840fd8ffc remving blank line 2024-05-02 02:21:59 -03:00
João Moura
0c717fb24a fixing long temr memory interpolation 2024-04-07 14:55:35 -03:00
João Moura
fd12609330 Adding new test for crew memory 2024-04-04 15:29:45 -03:00
João Moura
35f6156ed4 checking crew before using memory 2024-04-03 05:03:43 -03:00
João Moura
f85bf00409 Adding long term, short term, entity and contextual memory 2024-04-01 10:45:17 -03:00
João Moura
d73dd08ef4 Adding custom caching 2024-04-01 10:43:05 -03:00
João Moura
81d5fe6fc6 Ability to disable cache at agent and crew level 2024-04-01 10:43:05 -03:00
GabeKoga
eb98b8efad feature: human input per task (#395)
* feature: human input per task

* Update executor.py

* Update executor.py

* Update executor.py

* Update executor.py

* Update executor.py

* feat: change human input for unit testing
added documentation and unit test

* Create test_agent_human_input.yaml

add yaml for test

---------

Co-authored-by: João Moura <joaomdmoura@gmail.com>
2024-04-01 10:04:56 -03:00
João Moura
8468445e1d Update inner tool usage logic to support both regular and function calling 2024-03-03 20:54:15 -03:00
João Moura
b6badbaf54 changing method naming to increment 2024-02-28 03:44:23 -03:00
João Moura
5d682a5d6c Adding initial formatting error counting and token counter 2024-02-28 03:44:23 -03:00
João Moura
c099776962 Improving inner prompts 2024-02-20 17:53:30 -03:00
João Moura
9bd1dd4d01 bug fixing 2024-02-20 10:40:16 -03:00
João Moura
3cfc8dd4e0 Adding new tool usage and parsing logic 2024-02-19 22:43:10 -03:00
João Moura
430eb23448 Cutting new version with tool ussage bug fix 2024-02-15 23:19:12 -03:00
João Moura
55c0c186d1 adding function calling llm support 2024-02-13 02:57:12 -08:00
João Moura
21d1168943 Adding ability to remember instruction after using too many tools 2024-02-10 12:53:02 -08:00
João Moura
5a102251cf Revamping tool usage 2024-02-10 10:36:34 -08:00
João Moura
a9da07dc5d include agentFinish as part of step callback 2024-02-09 02:00:41 -08:00
João Moura
43542f226b adding support for step_callback 2024-02-08 23:56:13 -08:00
João Moura
c5839b215e adding agent step callback 2024-02-08 23:01:30 -08:00
João Moura
2c86f0d07c updating RPM and max_inter logic 2024-02-05 23:14:22 -08:00
Guilherme Vieira
e0d97b9916 Fix static typing errors (#187)
Co-authored-by: João Moura <joaomdmoura@gmail.com>
2024-01-29 19:52:14 -03:00
João Moura
e27dd53c78 Add RPM control to both agents and crews (#133)
* moving file into utilities
* creating Logger and RPMController
* Adding support for RPM to agents and crew
2024-01-14 00:22:11 -03:00
João Moura
1fc806161f Adding support for Crew throttling using RPM (#124)
* Add translations
* fixing translations
* Adding support for Crew throttling with RPM
2024-01-13 11:20:30 -03:00
João Moura
b487136878 Adding support for translations (#120)
Add translations support
2024-01-12 14:49:36 -03:00
João Moura
70072b4e40 Revamp max iteration Logic (#111)
This now will allow to add a max_inter option to agents while also making sure to force the agent to give it's best final answer before running out of it's max_inter.
2024-01-11 12:32:54 -03:00