Files
crewAI/lib/cli/src/crewai_cli/create_json_crew.py
João Moura 2b56dab813 feat(cli): pull latest LLM models dynamically in the crew wizard (#6462)
* feat(cli): pull latest LLM models dynamically in the crew wizard

The JSON-crew creation wizard hardcoded a short model list per provider,
which goes stale as vendors ship new models every few weeks. Add a
three-tier resolver that prefers live data and falls back to a curated list.

- New `model_catalog.get_provider_models(provider, fallback)`:
  1. Vendor API (openai/anthropic/gemini/groq/cerebras/ollama) when the
     provider key is already in the environment — the only reliably-fresh
     source (real release dates / display names).
  2. Curated hardcoded fallback — hand-verified, used when no key is set.
  3. LiteLLM feed — only for providers with no curated list; it lags real
     releases, so it must never preempt the curated fallback.
- Rank by date/version parsed from model ids, humanize labels, 6h cache,
  short timeouts, silent fallback on any error.
- Wire it into `create_json_crew._select_model()` (picker only).
- Refresh the curated fallback against each vendor's official model docs
  (Anthropic Fable 5 / Opus 4.8 / Sonnet 5; OpenAI GPT-5.5(+pro); Gemini
  3.5 Flash / 3.1 Pro preview / 3 Flash preview; Groq Llama 4 / GPT-OSS).
- Tests for ranking, chat filtering, caching, and the tier order (17 tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): address model_catalog review findings

- Key the Ollama catalog cache by its base URL so a changed OLLAMA_API_BASE /
  API_BASE no longer serves the previous host's models for up to the TTL.
- Negatively cache the curated fallback after a failed/empty fetch (short
  _NEGATIVE_TTL) so the picker doesn't repeat a timeout-prone vendor/LiteLLM
  request on every call — most impactful for a down local Ollama server.
- Guard _read_catalog_cache / _write_catalog_cache against a non-dict cache
  root (corrupt JSON array no longer raises AttributeError).
- Replace the two empty `except OSError: pass` blocks with
  contextlib.suppress(OSError) plus an explanatory comment (CodeQL empty-except).
- Tests: negative cache, base-keyed Ollama cache, corrupt-cache no-crash (20 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): guard null litellm_provider and paginate Gemini models

- _from_litellm: coerce a present-but-null `litellm_provider` before string
  ops so it's skipped instead of raising AttributeError (keeps the documented
  "never raises" contract).
- _fetch_gemini: walk models.list pages via nextPageToken (bounded to 10) —
  the API is paginated and not guaranteed newest-first, so a single page could
  drop models the ranking should consider.
- Tests for both (22 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* ci: ignore nltk PYSEC-2026-597 in pip-audit (no fix, not reachable)

pip-audit newly flags nltk 3.9.4 for PYSEC-2026-597 (CVE-2026-12243), a path
traversal via percent-encoded `..%2f` in nltk.data.load()/find(). It affects
all nltk versions <=3.9.4 with no patched release, so it can't be resolved by a
version bump — same situation as the already-ignored PYSEC-2026-97.

nltk is a transitive dependency (unstructured[local-inference, all-docs] in
crewai-tools) used for text tokenization; we never pass untrusted resource
URLs/paths to nltk.data, so the traversal is not reachable. Add it to the
curated --ignore-vuln list with a justification, matching the existing pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): address second Cursor review round on model_catalog

- Cache ignores new API keys: include API-key presence in the cache key
  (`<provider>#key|#nokey`), so a key added after a no-key/negative-cached
  lookup triggers a fresh live fetch instead of serving the stale fallback.
- Bad LiteLLM cache crashes picker: `_from_litellm` now requires a dict from
  `_load_litellm_data` (a non-mapping JSON root is skipped, not `.items()`'d).
- Stale LiteLLM refetch loop: memoize the feed load once per process
  (`_litellm_memo` + `_reset_litellm_memo` test hook) so repeated uncurated-
  provider lookups don't each re-attempt a timed download when offline.
- Tests: new-key bypass, corrupt-litellm-cache no-crash, one-fetch-per-process
  (25 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): keep partial Gemini results on later-page fetch error

_fetch_gemini paginates; a network/HTTP error on page 2+ previously raised out
through _from_vendor, discarding models already parsed from earlier pages and
forcing the curated fallback. Catch per-page fetch errors and return the
partial set instead (a first-page failure still yields an empty list -> fallback).
Test: test_vendor_gemini_keeps_partial_on_later_page_error (26 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): don't let an invalid fresh LiteLLM cache block download

_fetch_litellm_data treated any truthy JSON root in a fresh provider_cache.json
as the feed and returned it, so a non-mapping root (e.g. a JSON array) was
memoized and the tier never re-downloaded until the file aged out — leaving
uncurated providers with an empty picker despite a recoverable cache. Only
short-circuit on a usable dict; otherwise fall through to the download.
Test renamed to test_invalid_litellm_cache_falls_through_to_download (asserts
recovery via refetch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): honor OLLAMA_HOST and treat empty vendor list as authoritative

- Ollama host mismatch: _ollama_base now also reads OLLAMA_HOST (the Ollama
  runtime convention) after OLLAMA_API_BASE/API_BASE, normalizing a scheme-less
  value (e.g. "127.0.0.1:11434" -> "http://127.0.0.1:11434"), so users who set
  only OLLAMA_HOST see models from the server the crew will actually use.
- Empty vendor list: a successful vendor fetch returning no models is now
  authoritative instead of collapsing to the curated fallback. A reachable
  Ollama with nothing installed yields an empty list (the picker prompts for
  manual entry) rather than offering hardcoded models that aren't installed; a
  failed fetch still falls back. _from_vendor now returns [] on success-empty
  and None only when the tier is unavailable.
- Tests: ollama empty->manual, ollama down->fallback, OLLAMA_HOST resolution
  (29 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): Gemini first-page failure falls back instead of showing empty

Interaction from the prior two fixes: _fetch_gemini swallowed a first-page
error and returned [], which _from_vendor reported as a successful-empty result
and get_provider_models treated as authoritative — skipping the curated Gemini
fallback and jumping to manual entry. Now a first-page failure (nothing gathered
yet) re-raises so _from_vendor returns None and the curated list is used; a
later-page failure still keeps the partial results.
Test: test_vendor_gemini_first_page_error_uses_fallback (30 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): Gemini GOOGLE_API_KEY + Ollama recovery not blocked by cache

- Gemini ignores GOOGLE_API_KEY: _PROVIDER_KEY_ENV now maps each provider to a
  tuple of accepted env vars; Gemini accepts GEMINI_API_KEY or GOOGLE_API_KEY
  (matching crewai's own Gemini provider). A new _provider_api_key() resolver
  is used by both _from_vendor and the cache key, so a GOOGLE_API_KEY user gets
  the live models API instead of the stale curated fallback.
- Ollama recovery blocked by cache: skip the negative (fallback) cache for
  Ollama. It's a local, fast-failing server, so re-probing each call is cheap
  and lets the picker pick up real installed models as soon as the server comes
  up, instead of serving suggestions for the negative-cache TTL.
- Tests: GOOGLE_API_KEY live fetch, Ollama down->recover (32 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* style(cli): ruff format model_catalog.py

Add the blank line ruff format expects after _provider_api_key; no behavior
change. Fixes the lint-run `ruff format --check lib/` step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): don't treat 'search' substring as a non-chat model marker

The 'search' entry in _NON_CHAT_MARKERS matched anywhere in a model id, dropping
legitimate completion models like gpt-4o-search-preview and anything containing
'research' (e.g. o3-deep-research, since 'search' is a substring). Remove it;
the remaining markers (embedding/audio/image/moderation/etc.) still filter
genuine non-chat models. Test: test_search_substring_not_treated_as_non_chat (33).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* Revert "ci: ignore nltk PYSEC-2026-597 in pip-audit"

Do not suppress an unpatched security advisory to make CI green. Remove
PYSEC-2026-597 from the pip-audit ignore list; leave the scan failing so it
keeps surfacing the nltk path traversal (CVE-2026-12243). This PR should not be
merged until nltk ships a fix (or the vulnerable transitive dep is otherwise
resolved).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): exclude fine-tuned models and checkpoints from the picker

With a live OPENAI_API_KEY, /v1/models returns the user's fine-tunes and
training checkpoints (ft:..., ...:ckpt-step-N). Their recent `created`
timestamps ranked them above the base models and filled every slot, so the
picker showed a wall of `ft:gpt-4o-mini-...:crewai::...` with mangled labels and
no foundation models at all. Skip fine-tunes/checkpoints in the OpenAI-shaped
fetcher so clean base models surface; a user who wants a fine-tune can still
enter it via the picker's "Other" option. Test:
test_openai_excludes_fine_tunes_and_checkpoints (34 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

* fix(cli): cleaner model labels + filter Ollama non-chat models

Anthropic/Gemini already use vendor display names; OpenAI/Groq/Cerebras/Ollama
fall to _humanize for anything outside the curated map, which produced mediocre
labels ("GPT Oss 120b", "qwen3 32b", "Deepseek r1", "llama3.3:70b").

Improve _humanize:
- split on ':' too (Ollama tags: llama3.3:70b -> "Llama3.3 70B")
- uppercase size suffixes (70b -> 70B), acronyms OSS/IT, brand casing
  (DeepSeek, ChatGPT, QwQ)
- capitalize the leading letter of fused family+version tokens (qwen3 -> Qwen3)
  while preserving OpenAI o-series lowercase (o3, o1-mini)

Also fix _fetch_ollama: /api/tags lists everything installed, so filter
non-chat (embedding) and fine-tune entries the same way the other tiers do.

Tests: expanded test_humanize + test_ollama_excludes_embedding_models (35 total).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBYGqJHC2TMC6fonFziuuh

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>
2026-07-06 20:49:52 -03:00

988 lines
33 KiB
Python

"""Scaffold a new JSON-first crew project."""
from __future__ import annotations
import json
from pathlib import Path
import re
import sys
from typing import Any
import click
from rich.console import Console
from rich.text import Text
from crewai_cli.constants import ENV_VARS
from crewai_cli.git import initialize_if_git_available
from crewai_cli.model_catalog import get_provider_models
from crewai_cli.tui_picker import pick_many, pick_one
from crewai_cli.utils import (
enable_prompt_line_editing,
is_dmn_mode_enabled,
load_env_vars,
render_template,
write_env_file,
)
from crewai_cli.version import get_crewai_tools_dependency
# ── Provider / model data ───────────────────────────────────────
_PROVIDERS: list[tuple[str, str]] = [
("openai", "OpenAI"),
("anthropic", "Anthropic"),
("gemini", "Google Gemini"),
("groq", "Groq"),
("ollama", "Ollama"),
("bedrock", "AWS Bedrock"),
("azure", "Azure OpenAI"),
("nvidia_nim", "NVIDIA NIM"),
("huggingface", "Hugging Face"),
("cerebras", "Cerebras"),
("sambanova", "SambaNova"),
("watson", "IBM watsonx"),
]
# Curated offline fallback / label source. The picker prefers models pulled
# live from the vendor's own API via ``model_catalog.get_provider_models``;
# this list is the hand-verified backstop used when no API key is available.
# Keep entries to real, current model ids — last verified against each vendor's
# official model docs on 2026-07-05.
_PROVIDER_MODELS: dict[str, list[tuple[str, str]]] = {
"openai": [
("gpt-5.5", "GPT-5.5"),
("gpt-5.5-pro", "GPT-5.5 Pro"),
("gpt-5.4", "GPT-5.4"),
("gpt-5.4-mini", "GPT-5.4 Mini"),
("gpt-5.2", "GPT-5.2"),
("gpt-4.1", "GPT-4.1"),
],
"anthropic": [
("claude-fable-5", "Claude Fable 5"),
("claude-opus-4-8", "Claude Opus 4.8"),
("claude-sonnet-5", "Claude Sonnet 5"),
("claude-opus-4-7", "Claude Opus 4.7"),
("claude-haiku-4-5", "Claude Haiku 4.5"),
("claude-sonnet-4-6", "Claude Sonnet 4.6"),
],
"gemini": [
("gemini-3.5-flash", "Gemini 3.5 Flash"),
("gemini-3.1-pro-preview", "Gemini 3.1 Pro (preview)"),
("gemini-3-flash-preview", "Gemini 3 Flash (preview)"),
("gemini-2.5-pro", "Gemini 2.5 Pro"),
("gemini-2.5-flash", "Gemini 2.5 Flash"),
("gemini-2.5-flash-lite", "Gemini 2.5 Flash Lite"),
],
"groq": [
("meta-llama/llama-4-maverick-17b-128e-instruct", "Llama 4 Maverick"),
("meta-llama/llama-4-scout-17b-16e-instruct", "Llama 4 Scout"),
("openai/gpt-oss-120b", "GPT-OSS 120B"),
("qwen/qwen3-32b", "Qwen3 32B"),
("moonshotai/kimi-k2-instruct-0905", "Kimi K2"),
("llama-3.3-70b-versatile", "Llama 3.3 70B"),
],
"ollama": [
("llama3.3", "Llama 3.3"),
("qwen3", "Qwen 3"),
("deepseek-r1", "DeepSeek R1"),
("gpt-oss", "GPT-OSS"),
("gemma3", "Gemma 3"),
("mistral", "Mistral"),
],
}
_TEMPLATES_DIR = Path(__file__).parent / "templates" / "json_crew"
# ── Common tools for picker ────────────────────────────────────
_TOOL_CATEGORIES: list[tuple[str, list[tuple[str, str]]]] = [
(
"Search & Research",
[
("SerperDevTool", "Google search via Serper API"),
("BraveSearchTool", "Web search via Brave Search"),
("BraveWebSearchTool", "Focused Brave web search"),
("BraveNewsSearchTool", "Search current news with Brave"),
("BraveImageSearchTool", "Search images with Brave"),
("BraveVideoSearchTool", "Search videos with Brave"),
("BraveLocalPOIsTool", "Find local places with Brave"),
("BraveLocalPOIsDescriptionTool", "Describe local places with Brave"),
("BraveLLMContextTool", "Fetch Brave search context"),
("TavilySearchTool", "Web search via Tavily"),
("TavilyResearchTool", "Run Tavily research"),
("TavilyGetResearchTool", "Retrieve Tavily research results"),
("TavilyExtractorTool", "Extract content with Tavily"),
("EXASearchTool", "Semantic web search via Exa"),
("ExaSearchTool", "Semantic web search via Exa"),
("LinkupSearchTool", "Web search via Linkup"),
("SerpApiGoogleSearchTool", "Google search via SerpApi"),
("SerpApiGoogleShoppingTool", "Google Shopping via SerpApi"),
("SerplyWebSearchTool", "Web search via Serply"),
("SerplyNewsSearchTool", "News search via Serply"),
("SerplyScholarSearchTool", "Scholar search via Serply"),
("SerplyJobSearchTool", "Job search via Serply"),
("SerplyWebpageToMarkdownTool", "Convert webpages with Serply"),
("ParallelSearchTool", "Run parallel web searches"),
("BrightDataSearchTool", "Search with Bright Data"),
("GithubSearchTool", "Search GitHub repositories"),
("ArxivPaperTool", "Search arXiv academic papers"),
],
),
(
"Web Scraping",
[
("ScrapeWebsiteTool", "Extract content from a URL"),
("ScrapeElementFromWebsiteTool", "Extract page elements from a URL"),
("FirecrawlScrapeWebsiteTool", "Scrape with Firecrawl"),
("FirecrawlCrawlWebsiteTool", "Crawl a website with Firecrawl"),
("FirecrawlSearchTool", "Search with Firecrawl"),
("SeleniumScrapingTool", "Browser-based scraping"),
("JinaScrapeWebsiteTool", "Scrape with Jina"),
("ScrapegraphScrapeTool", "AI-powered page scraping"),
("SerperScrapeWebsiteTool", "Scrape pages with Serper"),
("BrowserbaseLoadTool", "Load web pages with Browserbase"),
("HyperbrowserLoadTool", "Load web pages with Hyperbrowser"),
("MultiOnTool", "Control web workflows with MultiOn"),
("SpiderTool", "Crawl websites with Spider"),
("StagehandTool", "Browser automation with Stagehand"),
("BrightDataWebUnlockerTool", "Unlock websites with Bright Data"),
("BrightDataDatasetTool", "Fetch Bright Data datasets"),
("WebsiteSearchTool", "RAG search on a website"),
],
),
(
"File & Document",
[
("DirectoryReadTool", "List directory contents"),
("DirectorySearchTool", "Search directory contents"),
("FileReadTool", "Read local files"),
("FileWriterTool", "Write to local files"),
("FileCompressorTool", "Compress local files"),
("CSVSearchTool", "Search within CSV files"),
("PDFSearchTool", "Search within PDF files"),
("DOCXSearchTool", "Search within DOCX files"),
("MDXSearchTool", "Search within MDX files"),
("JSONSearchTool", "Search within JSON files"),
("TXTSearchTool", "Search within text files"),
("XMLSearchTool", "Search within XML files"),
("OCRTool", "Extract text with OCR"),
("YoutubeVideoSearchTool", "Search within YouTube videos"),
("YoutubeChannelSearchTool", "Search within YouTube channels"),
],
),
(
"Code & Data",
[
("CodeDocsSearchTool", "Search code documentation"),
("RagTool", "RAG over custom data sources"),
("NL2SQLTool", "Natural language to SQL queries"),
("DatabricksQueryTool", "Query Databricks data"),
("SingleStoreSearchTool", "Search SingleStore data"),
],
),
(
"Cloud & Storage",
[
("S3ReaderTool", "Read objects from Amazon S3"),
("S3WriterTool", "Write objects to Amazon S3"),
("BedrockInvokeAgentTool", "Invoke an Amazon Bedrock agent"),
("BedrockKBRetrieverTool", "Retrieve from Bedrock knowledge bases"),
],
),
(
"Sandbox & Automation",
[
("E2BExecTool", "Run commands in E2B"),
("E2BFileTool", "Manage files in E2B"),
("E2BPythonTool", "Run Python in E2B"),
("DaytonaExecTool", "Run commands in Daytona"),
("DaytonaFileTool", "Manage files in Daytona"),
("DaytonaPythonTool", "Run Python in Daytona"),
("GenerateCrewaiAutomationTool", "Generate CrewAI automations"),
],
),
(
"AI & Vision",
[
("DallETool", "Generate images with DALL-E"),
("VisionTool", "Analyze images with vision models"),
("AIMindTool", "Connect to MindStudio agents"),
("PatronusEvalTool", "Evaluate output with Patronus"),
("PatronusLocalEvaluatorTool", "Run local Patronus evaluations"),
],
),
]
_FLAT_TOOLS: list[tuple[str, str]] = [
tool for _cat, tools in _TOOL_CATEGORIES for tool in tools
]
_COMMON_TOOL_ORDER = [
"SerperDevTool",
"ScrapeWebsiteTool",
"DirectoryReadTool",
"FileReadTool",
"FileWriterTool",
]
_ANSI_SEQUENCE_RE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]")
# ── Interactive wizard ─────────────────────────────────────────
def _prompt_text(
label: str,
default: str = "",
*,
spacing_before: bool = True,
) -> str:
if spacing_before:
click.echo()
prompt = click.style(f" {label}", fg="cyan")
if default:
prompt += f" [{default}]"
prompt += click.style(" > ", fg="bright_white")
try:
value = input(_readline_safe_prompt(prompt))
except (KeyboardInterrupt, EOFError):
raise click.Abort() from None
if not value and default:
value = default
return value.strip()
def _readline_safe_prompt(prompt: str) -> str:
if not sys.stdin.isatty():
return prompt
try:
import readline # noqa: F401
except ImportError:
return prompt
return _ANSI_SEQUENCE_RE.sub(lambda match: f"\001{match.group(0)}\002", prompt)
def _confirm(label: str, default: bool = False) -> bool:
click.echo()
return click.confirm(
click.style(f" {label}", fg="cyan"),
default=default,
prompt_suffix=click.style(" > ", fg="bright_white"),
)
def _success(message: str, *, bold: bool = False, dim: bool = False) -> None:
click.echo()
click.secho(f"{message}", fg="green", bold=bold, dim=dim)
def _highlight_placeholders(text: str) -> Text:
highlighted = Text(text, style="dim")
highlighted.highlight_regex(r"\{[A-Za-z_][A-Za-z0-9_]*\}", style="bold cyan")
return highlighted
def _show_interpolation_hint(kind: str) -> None:
console = Console()
console.print(
_highlight_placeholders(
" Tip: Use {placeholder} for dynamic values you want to change later."
)
)
def _tool_label(name: str, description: str) -> str:
return f"{description:<48s} {name}"
def _tool_category_label(category: str) -> str:
return f"── {category} ──"
def _category_row_label(
category: str, tools: list[tuple[str, str]], selected: set[str], expanded: bool
) -> str:
"""Render an accordion category row with tool/selection counts."""
marker = "" if expanded else ""
sel_count = sum(1 for name, _desc in tools if name in selected)
suffix = f"{len(tools)} tools"
if sel_count:
suffix += f", {sel_count} selected"
return f"{marker} {category} ({suffix})"
def _select_tools() -> list[str]:
"""Accordion tool picker.
Common tools are always visible at the top; every other category shows
as a single expandable row. Expanding one category collapses the others.
Selections persist while expanding/collapsing.
"""
tools_by_name = {name: desc for name, desc in _FLAT_TOOLS}
common_tools = [
(name, tools_by_name[name])
for name in _COMMON_TOOL_ORDER
if name in tools_by_name
]
common_tool_names = {name for name, _desc in common_tools}
categories: list[tuple[str, list[tuple[str, str]]]] = []
for category, category_tools in _TOOL_CATEGORIES:
remaining_tools = [
(name, desc)
for name, desc in category_tools
if name not in common_tool_names
]
if remaining_tools:
categories.append((category, remaining_tools))
selected: set[str] = set()
expanded: str | None = None
focus_category: str | None = None
while True:
labels: list[str] = []
tool_by_index: dict[int, str] = {}
separator_indices: set[int] = set()
action_indices: set[int] = set()
category_by_index: dict[int, str] = {}
preselected: set[int] = set()
initial_cursor: int | None = None
separator_indices.add(len(labels))
labels.append(_tool_category_label("Common tools"))
for name, desc in common_tools:
if name in selected:
preselected.add(len(labels))
tool_by_index[len(labels)] = name
labels.append(_tool_label(name, desc))
for category, category_tools in categories:
row = len(labels)
action_indices.add(row)
category_by_index[row] = category
is_expanded = category == expanded
if category == focus_category:
initial_cursor = row
labels.append(
_category_row_label(category, category_tools, selected, is_expanded)
)
if is_expanded:
for name, desc in category_tools:
if name in selected:
preselected.add(len(labels))
tool_by_index[len(labels)] = name
labels.append(_tool_label(name, desc))
indices, action = pick_many(
"Tools (space to toggle, enter to confirm):",
labels,
action_indices=action_indices,
separator_indices=separator_indices,
preselected=preselected,
initial_cursor=initial_cursor,
)
# Carry over toggles made on this screen; tools not visible in this
# render keep their previous state.
visible = set(tool_by_index.values())
chosen = {tool_by_index[i] for i in indices if i in tool_by_index}
selected = (selected - visible) | chosen
if action is None:
break
toggled = category_by_index.get(action)
focus_category = toggled
expanded = None if toggled == expanded else toggled
ordered = [name for name, _desc in common_tools] + [
name for _cat, cat_tools in categories for name, _desc in cat_tools
]
return [name for name in ordered if name in selected]
def _wizard_agent(
agent_num: int,
existing_names: list[str],
skip_provider: bool = False,
last_llm: str | None = None,
preset_llm: str | None = None,
) -> dict[str, Any] | None:
"""Interactive wizard for one agent. Returns agent dict or None if skipped."""
click.echo()
click.secho(f" Agent {agent_num}", fg="cyan", bold=True)
role = _prompt_text("Role", spacing_before=False)
if not role:
return None
name_default = role.lower().replace(" ", "_")[:30]
name_default = re.sub(r"[^a-z0-9_]", "", name_default)
if not name_default:
# Roles made only of symbols would otherwise produce an empty slug
# and an invalid agents/.jsonc file name.
name_default = f"agent_{agent_num}"
while name_default in existing_names:
name_default += "_2"
goal = _prompt_text("Goal", spacing_before=False)
backstory = _prompt_text("Backstory", spacing_before=False)
# LLM model
if preset_llm:
llm = preset_llm
_success(llm)
elif skip_provider:
llm = last_llm or "openai/gpt-4o"
elif last_llm:
reuse_labels = [
f"Same as before ({last_llm})",
"Choose a different model",
]
r_idx = pick_one("LLM:", reuse_labels)
if r_idx == 1:
llm = _select_model()
else:
llm = last_llm
_success(llm)
else:
llm = _select_model()
tools = _select_tools()
if tools:
_success(f"{len(tools)} tool{'s' if len(tools) != 1 else ''}")
else:
_success("No tools", dim=True)
# Planning
planning = _confirm("Enable step-by-step planning?", default=False)
# Allow delegation
allow_delegation = _confirm("Allow delegation to other agents?", default=False)
return {
"name": name_default,
"role": role,
"goal": goal,
"backstory": backstory,
"llm": llm,
"tools": tools,
"planning": planning,
"allow_delegation": allow_delegation,
}
def _wizard_task(
task_num: int,
agent_names: list[str],
prior_task_names: list[str],
) -> dict[str, Any] | None:
"""Interactive wizard for one task. Returns task dict or None if skipped."""
click.echo()
click.secho(f" Task {task_num}", fg="cyan", bold=True)
description = _prompt_text("Description", spacing_before=False)
if not description:
return None
# Auto-generate name from first few words of description
words = description.lower().split()[:4]
base = re.sub(r"[^a-z0-9_]", "", "_".join(words))
name = f"{base}_task" if base else f"task_{task_num}"
while name in prior_task_names:
name += "_2"
expected_output = _prompt_text("Expected output", spacing_before=False)
# Agent assignment
if len(agent_names) == 1:
assigned_agent = agent_names[0]
else:
a_idx = pick_one("Assign to agent:", agent_names)
while a_idx < 0:
click.secho(" Every task needs an agent — pick one to continue.", dim=True)
a_idx = pick_one("Assign to agent:", agent_names)
assigned_agent = agent_names[a_idx]
_success(f"Agent: {assigned_agent}")
# Context dependencies
context: list[str] = []
if prior_task_names:
ctx_indices = pick_many(
"Context from prior tasks (space to toggle):",
[*prior_task_names, "None"],
)
context = [
prior_task_names[i] for i in ctx_indices if i < len(prior_task_names)
]
if context:
_success(f"Context: {', '.join(context)}")
return {
"name": name,
"description": description,
"expected_output": expected_output,
"agent": assigned_agent,
"context": context,
}
def _wizard_agents_and_tasks(
skip_provider: bool = False,
default_llm: str | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
"""Run the full interactive wizard. Returns (agents, tasks, crew_settings)."""
agents: list[dict[str, Any]] = []
tasks: list[dict[str, Any]] = []
# ── Step 1: Agents ──
click.echo()
click.secho(" Step 1/3 — Agents", fg="cyan", bold=True)
click.secho(" Define the AI agents in your crew.", dim=True)
_show_interpolation_hint("agents")
while True:
last_llm = agents[-1]["llm"] if agents else None
agent = _wizard_agent(
agent_num=len(agents) + 1,
existing_names=[a["name"] for a in agents],
skip_provider=skip_provider,
last_llm=last_llm,
preset_llm=default_llm if not agents else None,
)
if agent is None and not agents:
click.secho(" Need at least one agent.", fg="yellow")
continue
if agent is not None:
agents.append(agent)
_success(f"{agent['role']} added", bold=True)
if not _confirm("Add another agent?", default=False):
break
# ── Step 2: Tasks ──
click.echo()
click.secho(" Step 2/3 — Tasks", fg="cyan", bold=True)
click.secho(" Define what your agents should do.", dim=True)
_show_interpolation_hint("tasks")
agent_names = [a["name"] for a in agents]
task_names: list[str] = []
while True:
task = _wizard_task(
task_num=len(tasks) + 1,
agent_names=agent_names,
prior_task_names=task_names,
)
if task is None and not tasks:
click.secho(" Need at least one task.", fg="yellow")
continue
if task is not None:
tasks.append(task)
task_names.append(task["name"])
_success(f"Task {len(tasks)} added", bold=True)
if not _confirm("Add another task?", default=False):
break
# ── Step 3: Settings ──
click.echo()
click.secho(" Step 3/3 — Settings", fg="cyan", bold=True)
process = "sequential"
memory = _confirm("Enable crew memory?", default=True)
crew_settings = {
"process": process,
"memory": memory,
"inputs": {},
}
return agents, tasks, crew_settings
def _default_agents_and_tasks(
default_llm: str | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
"""Return deterministic scaffold data for non-interactive project creation."""
llm = default_llm or "openai/gpt-4o"
agents = [
{
"name": "researcher",
"role": "Senior Researcher",
"goal": "Research the requested topic and identify useful findings.",
"backstory": (
"You are an experienced researcher who finds relevant information "
"and presents it clearly."
),
"llm": llm,
"tools": [],
"planning": False,
"allow_delegation": False,
}
]
tasks = [
{
"name": "research_task",
"description": "Research current AI trends and write a concise summary.",
"expected_output": "A concise markdown report with key findings.",
"agent": "researcher",
"context": [],
}
]
crew_settings = {
"process": "sequential",
"memory": True,
"inputs": {},
}
return agents, tasks, crew_settings
# ── JSONC generation from wizard data ──────────────────────────
def _agent_to_jsonc(agent: dict[str, Any]) -> str:
"""Convert agent wizard data to JSONC string with comments."""
has_planning = agent["planning"]
settings_block = _render_json_crew_template(
"agent_settings.jsonc",
{
"allow_delegation": "true" if agent["allow_delegation"] else "false",
"delegation_comma": "," if has_planning else "",
"planning_line": '"planning": true'
if has_planning
else '// "planning": false',
},
)
return _render_json_crew_template(
"agent.jsonc",
{
"role_json": json.dumps(agent["role"]),
"goal_json": json.dumps(agent["goal"]),
"backstory_json": json.dumps(agent["backstory"]),
"llm_json": json.dumps(agent["llm"]),
"tools_json": json.dumps(agent["tools"]),
"settings_block": settings_block,
},
)
def _task_to_json_fragment(task: dict[str, Any]) -> str:
"""Convert task wizard data to a JSON-like fragment for embedding in crew JSONC."""
has_context = bool(task.get("context"))
has_output_file = bool(task.get("output_file"))
context_block = ""
output_file_block = ""
if has_context:
context_block = (
"\n\n"
" // Task outputs used as context\n"
f' "context": {json.dumps(task["context"])}'
f"{',' if has_output_file else ''}"
)
if has_output_file:
output_file_block = (
"\n\n"
" // Save output to a file\n"
f' "output_file": {json.dumps(task["output_file"])}'
)
return _render_json_crew_template(
"task.jsonc",
{
"name_json": json.dumps(task["name"]),
"description_json": json.dumps(task["description"]),
"expected_output_json": json.dumps(task["expected_output"]),
"agent_json": json.dumps(task["agent"]),
"agent_comma": "," if has_context or has_output_file else "",
"context_block": context_block,
"output_file_block": output_file_block,
},
)
def _crew_to_jsonc(
name: str,
agents: list[dict[str, Any]],
tasks: list[dict[str, Any]],
settings: dict[str, Any],
) -> str:
"""Generate the full crew.jsonc from wizard data."""
agent_names_json = json.dumps([a["name"] for a in agents])
tasks_fragments = ",\n".join(_task_to_json_fragment(t) for t in tasks)
inputs_json = json.dumps(settings.get("inputs", {}), indent=4)
# Re-indent inputs to 4-space
inputs_lines = inputs_json.split("\n")
if len(inputs_lines) > 1:
inputs_json = (
inputs_lines[0] + "\n" + "\n".join(" " + line for line in inputs_lines[1:])
)
memory = "true" if settings.get("memory") else "false"
return _render_json_crew_template(
"crew.jsonc",
{
"name_json": json.dumps(name),
"agent_names_json": agent_names_json,
"tasks_fragments": tasks_fragments,
"process_json": json.dumps(settings.get("process", "sequential")),
"memory": memory,
"manager_agent_name": agents[0]["name"],
"inputs_json": inputs_json,
},
)
# ── Model selection ─────────────────────────────────────────────
def _select_model() -> str:
"""Two-step arrow-key selection: provider, then model."""
provider_labels = [label for _, label in _PROVIDERS]
provider_labels.append("Other (enter manually)")
p_idx = pick_one("LLM Provider:", provider_labels)
if p_idx < 0:
return "openai/gpt-4o"
if p_idx == len(_PROVIDERS):
custom: str = click.prompt(
click.style(" Enter model (provider/model)", fg="cyan"),
type=str,
prompt_suffix=click.style(" > ", fg="bright_white"),
)
return custom.strip()
provider_key, provider_name = _PROVIDERS[p_idx]
click.secho(f"{provider_name}", fg="green")
# Prefer the latest models pulled live from the vendor / LiteLLM; the
# curated ``_PROVIDER_MODELS`` entry is the offline fallback and label source.
models = get_provider_models(provider_key, _PROVIDER_MODELS.get(provider_key, []))
if not models:
custom = click.prompt(
click.style(f" Enter model name for {provider_key}/", fg="cyan"),
type=str,
prompt_suffix=click.style(" > ", fg="bright_white"),
)
return f"{provider_key}/{custom.strip()}"
model_labels = [f"{label} ({model_id})" for model_id, label in models]
model_labels.append("Other (enter model name)")
m_idx = pick_one(f"{provider_name} Model:", model_labels)
if m_idx < 0:
return f"{provider_key}/{models[0][0]}"
if m_idx == len(models):
custom = click.prompt(
click.style(f" Enter model name for {provider_key}/", fg="cyan"),
type=str,
prompt_suffix=click.style(" > ", fg="bright_white"),
)
result = f"{provider_key}/{custom.strip()}"
else:
model_id = models[m_idx][0]
result = f"{provider_key}/{model_id}"
click.secho(f"{result}", fg="green")
return result
def _default_model_for_provider(provider: str | None) -> str | None:
"""Return the default provider/model string for a ``--provider`` value."""
if not provider:
return None
normalized = provider.strip().lower()
if not normalized:
return None
if "/" in normalized:
return normalized
models = _PROVIDER_MODELS.get(normalized)
if not models:
return None
return f"{normalized}/{models[0][0]}"
# ── Helpers ─────────────────────────────────────────────────────
def _render_json_crew_template(
template_name: str, replacements: dict[str, str] | None = None
) -> str:
return render_template(_TEMPLATES_DIR / template_name, replacements or {})
def _write_jsonc(path: Path, content: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
def _setup_env(folder_path: Path, llm_model: str) -> None:
"""Prompt for API keys based on the selected provider."""
click.echo()
env_vars = load_env_vars(folder_path)
env_vars["MODEL"] = llm_model
provider = llm_model.split("/")[0] if "/" in llm_model else llm_model
if provider in ENV_VARS:
for details in ENV_VARS[provider]:
if details.get("default", False):
for key, value in details.items():
if key not in ["prompt", "key_name", "default"]:
env_vars[key] = value
elif "key_name" in details:
api_key_value = click.prompt(
click.style(f" {details['prompt']}", fg="cyan"),
default="",
show_default=False,
prompt_suffix=click.style(" > ", fg="bright_white"),
)
if api_key_value.strip():
env_vars[details["key_name"]] = api_key_value
if env_vars:
write_env_file(folder_path, env_vars)
click.secho(" API keys and model saved to .env file", fg="green")
# ── Main ────────────────────────────────────────────────────────
def create_json_crew(
name: str,
provider: str | None = None,
skip_provider: bool = False,
) -> None:
"""Scaffold a new JSON-first crew project."""
import keyword
import shutil
dmn_mode = is_dmn_mode_enabled()
if not dmn_mode:
enable_prompt_line_editing()
name = name.rstrip("/")
if not name.strip():
raise ValueError("Project name cannot be empty")
folder_name = name.replace(" ", "_").replace("-", "_").lower()
folder_name = re.sub(r"[^a-zA-Z0-9_]", "", folder_name)
if not folder_name or folder_name[0].isdigit():
raise ValueError(
f"Project name '{name}' produces invalid folder name '{folder_name}'"
)
if keyword.iskeyword(folder_name):
raise ValueError(f"'{folder_name}' is a reserved Python keyword")
folder_path = Path(folder_name)
if folder_path.exists():
if dmn_mode:
raise click.ClickException(f"Folder {folder_name} already exists.")
if not click.confirm(f"Folder {folder_name} already exists. Override?"):
click.secho("Cancelled.", fg="yellow")
sys.exit(0)
shutil.rmtree(folder_path)
click.echo()
click.secho(f" Creating crew: {name}", fg="green", bold=True)
default_llm = _default_model_for_provider(provider)
if dmn_mode:
agents, tasks, crew_settings = _default_agents_and_tasks(default_llm)
else:
agents, tasks, crew_settings = _wizard_agents_and_tasks(
skip_provider=skip_provider,
default_llm=default_llm,
)
# Create directories
folder_path.mkdir(parents=True)
(folder_path / "agents").mkdir()
(folder_path / "tools").mkdir()
(folder_path / "skills").mkdir()
(folder_path / "knowledge").mkdir()
for agent in agents:
_write_jsonc(
folder_path / "agents" / f"{agent['name']}.jsonc",
_agent_to_jsonc(agent),
)
_write_jsonc(
folder_path / "crew.jsonc",
_crew_to_jsonc(name, agents, tasks, crew_settings),
)
# Write pyproject.toml
(folder_path / "pyproject.toml").write_text(
_render_json_crew_template(
"pyproject.toml",
{
"folder_name": folder_name,
"name": name,
"crewai_tools_dependency": get_crewai_tools_dependency(),
},
),
encoding="utf-8",
)
# Write .gitignore
(folder_path / ".gitignore").write_text(
_render_json_crew_template(".gitignore"),
encoding="utf-8",
)
# Write README
(folder_path / "README.md").write_text(
_render_json_crew_template("README.md", {"name": name}),
encoding="utf-8",
)
# Write knowledge placeholder
(folder_path / "knowledge" / "user_preference.txt").write_text(
_render_json_crew_template("knowledge/user_preference.txt"),
encoding="utf-8",
)
# Keep skills dir tracked by git
(folder_path / "skills" / ".gitkeep").write_text("", encoding="utf-8")
# Setup .env with API keys
if not skip_provider and not dmn_mode:
models = list({a["llm"] for a in agents})
for model in models:
_setup_env(folder_path, model)
initialize_if_git_available(folder_path)
click.echo()
click.secho(f" ✔ Crew {name} created successfully!", fg="green", bold=True)
click.echo()
click.secho(" Next steps:", bold=True)
click.echo()
click.echo(f" cd {folder_name}")
click.echo()
click.secho(" Run your crew:", fg="cyan")
click.echo(" crewai run")
click.echo()
click.secho(" Customize your crew:", fg="cyan")
click.echo(" agents/*.jsonc Define agent roles, goals, and LLMs")
click.echo(" crew.jsonc Configure tasks and optional input defaults")
click.echo(" tools/ Add custom tools (Python)")
click.echo()