mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-01-07 23:28:30 +00:00
Update LLM-Connections.md (#359)
Created a short documentation on how to use Llama2 locally with crewAI thanks to the help of Ollama. Co-authored-by: João Moura <joaomdmoura@gmail.com>
This commit is contained in:
@@ -42,7 +42,7 @@ example_agent = Agent(
|
||||
```
|
||||
|
||||
## Ollama Integration
|
||||
Ollama is preferred for local LLM integration, offering customization and privacy benefits. To integrate Ollama with CrewAI, set the appropriate environment variables as shown below. Note: Detailed Ollama setup is beyond this document's scope, but general guidance is provided.
|
||||
Ollama is preferred for local LLM integration, offering customization and privacy benefits. To integrate Ollama with CrewAI, set the appropriate environment variables as shown below.
|
||||
|
||||
### Setting Up Ollama
|
||||
- **Environment Variables Configuration**: To integrate Ollama, set the following environment variables:
|
||||
@@ -52,6 +52,70 @@ OPENAI_MODEL_NAME='openhermes' # Adjust based on available model
|
||||
OPENAI_API_KEY=''
|
||||
```
|
||||
|
||||
## Ollama Integration (ex. for using Llama 2 locally)
|
||||
1. [Download Ollama](https://ollama.com/download).
|
||||
2. After setting up the Ollama, Pull the Llama2 by typing following lines into the terminal ```ollama pull Llama2```.
|
||||
3. Create a ModelFile similar the one below in your project directory.
|
||||
```
|
||||
FROM llama2
|
||||
|
||||
# Set parameters
|
||||
|
||||
PARAMETER temperature 0.8
|
||||
PARAMETER stop Result
|
||||
|
||||
# Sets a custom system message to specify the behavior of the chat assistant
|
||||
|
||||
# Leaving it blank for now.
|
||||
|
||||
SYSTEM """"""
|
||||
```
|
||||
4. Create a script to get the base model, which in our case is llama2, and create a model on top of that with ModelFile above. PS: this will be ".sh" file.
|
||||
```
|
||||
#!/bin/zsh
|
||||
|
||||
# variables
|
||||
model_name="llama2"
|
||||
custom_model_name="crewai-llama2"
|
||||
|
||||
#get the base model
|
||||
ollama pull $model_name
|
||||
|
||||
#create the model file
|
||||
ollama create $custom_model_name -f ./Llama2ModelFile
|
||||
```
|
||||
5. Go into the directory where the script file and ModelFile is located and run the script.
|
||||
6. Enjoy your free Llama2 model that powered up by excellent agents from crewai.
|
||||
```
|
||||
from crewai import Agent, Task, Crew
|
||||
from langchain_openai import ChatOpenAI
|
||||
import os
|
||||
os.environ["OPENAI_API_KEY"] = "NA"
|
||||
|
||||
llm = ChatOpenAI(
|
||||
model = "crewai-llama2",
|
||||
base_url = "http://localhost:11434/v1")
|
||||
|
||||
general_agent = Agent(role = "Math Professor",
|
||||
goal = """Provide the solution to the students that are asking mathematical questions and give them the answer.""",
|
||||
backstory = """You are an excellent math professor that likes to solve math questions in a way that everyone can understand your solution""",
|
||||
allow_delegation = False,
|
||||
verbose = True,
|
||||
llm = llm)
|
||||
task = Task (description="""what is 3 + 5""",
|
||||
agent = general_agent)
|
||||
|
||||
crew = Crew(
|
||||
agents=[general_agent],
|
||||
tasks=[task],
|
||||
verbose=2
|
||||
)
|
||||
|
||||
result = crew.kickoff()
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
## HuggingFace Integration
|
||||
There are a couple of different ways you can use HuggingFace to host your LLM.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user