fix(tracing): keep tool and task outputs whole in exported spans (#7833)

* fix(tracing): keep tool and task outputs whole in exported spans

Every gen_ai.* content attribute was capped at 32 KiB, and a value over the
cap that was not a message array became an envelope holding a 4 KiB preview:
a 33 KB tool result reached the trace as 4 KB. Graders checking a summary
against the data a tool returned could not decide.

- DEFAULT_MAX_ATTR_BYTES is 384 KiB, sized to Wharf's 3,072,000-byte request
  limit: seven attributes at the bound still fit one request.
- Over the bound, a value loses only what is over it (the envelope preview
  fills the bound; a message's text is cut by the overshoot, not halved), with
  the existing <attr>.truncated / <attr>.original_size_bytes markers.
- Every other string attribute (crewai.task.output, crewai.mcp.tool_result,
  serialized flow state) gets the same bound and markers; before, one huge
  value made the span too large for Wharf and the exporter dropped it whole.
- An OpenTelemetry SDK attribute length limit, when set, lowers the bound, so
  the SDK (which cuts without a marker) never has anything left to cut.
  CREWAI_OTEL_MAX_ATTR_BYTES still replaces the bound; no new setting.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(tracing): honour SDK length limits as the SDK reads them; cut plain attributes plainly

- The span attribute length limit is read like SpanLimits: the span setting
  wins over the general one, an explicitly empty value means unlimited, and 0
  is a limit (every string cut to nothing, with crewAI's marker).
- Plain attributes (crewai.task.output, crewai.mcp.tool_result, flow state)
  keep their head up to the bound with the .truncated / .original_size_bytes
  markers; only gen_ai.* messages get message-aware truncation, so a
  role-bearing JSON list a task produced is never reshaped as a conversation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(tracing): clamp the attribute bound at the Wharf-safe ceiling; shrink a span that does not fit instead of dropping it

- CREWAI_OTEL_MAX_ATTR_BYTES and the OTel SDK span attribute length limit
  can lower the 384 KiB bound, never raise it. A higher
  CREWAI_OTEL_MAX_ATTR_BYTES is clamped with one warning naming the value,
  the ceiling and why (Wharf's 3,072,000-byte request). No new setting.
- GrantSpanExporter no longer drops a span whose attributes, each within its
  bound, add up to more than one request: the largest string attributes are
  cut by the overshoot, each with .truncated / .original_size_bytes, until it
  fits. A span whose excess is in its resource, scope or events is still
  skipped, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.github/workflows/vulnerability-scan.yml

@@ -117,6 +117,16 @@ jobs:
             # patched release; keep the ignore list in sync with
             # .pre-commit-config.yaml.
             --ignore-vuln GHSA-4j2p-28q2-5m79
+            # oauthlib <4.0.0: GHSA-xpv3-w29h-x7cv: timing side channel in the
+            # server-side PKCE code_verifier check (authorization_code grant).
+            # The fix, 4.0.0 (published 2026-09-28), is inside the 3-day
+            # exclude-newer cooldown. Transitive via chromadb -> kubernetes ->
+            # requests-oauthlib, which uses oauthlib as an OAuth client only;
+            # CrewAI never runs an oauthlib authorization server.
+            # TODO: drop this ignore and run `uv lock --upgrade-package oauthlib`
+            # once 4.0.0 clears the cooldown (after 2026-10-01 06:01 UTC); keep
+            # the ignore list in sync with .pre-commit-config.yaml.
+            --ignore-vuln GHSA-xpv3-w29h-x7cv
           )
           uv run pip-audit "${pip_audit_args[@]}"
         continue-on-error: true

.pre-commit-config.yaml

@@ -31,6 +31,7 @@ repos:
         # Keep this ignore list in sync with .github/workflows/vulnerability-scan.yml.
         # TODO: drop --ignore-vuln GHSA-8mgp-746c-j5xp when bumping nltk past 3.10.3.
         # TODO: drop --ignore-vuln GHSA-4j2p-28q2-5m79 when bumping accelerate past 1.14.0.
+        # TODO: drop --ignore-vuln GHSA-xpv3-w29h-x7cv when bumping oauthlib to 4.0.0 (after 2026-10-01 06:01 UTC).
         entry: >-
           bash -c 'case "$OSTYPE" in msys*|cygwin*|win32*) source .venv/Scripts/activate ;; *) source .venv/bin/activate ;; esac && uv run pip-audit --skip-editable
           --ignore-vuln PYSEC-2024-277
@@ -63,7 +64,8 @@ repos:
           --ignore-vuln GHSA-36p7-vc44-83pf
           --ignore-vuln GHSA-xph7-9rjv-w5fr
           --ignore-vuln GHSA-8mgp-746c-j5xp
-          --ignore-vuln GHSA-4j2p-28q2-5m79' --
+          --ignore-vuln GHSA-4j2p-28q2-5m79
+          --ignore-vuln GHSA-xpv3-w29h-x7cv' --
         language: system
         pass_filenames: false
         stages: [pre-push, manual]

lib/cli/pyproject.toml

@@ -15,7 +15,7 @@ dependencies = [
     "appdirs~=1.4.4",
     "cryptography>=42.0",
     "httpx~=0.28.1",
-    "pyjwt>=2.13.0,<3",
+    "pyjwt>=2.15.0,<3",
     "rich>=13.7.1",
     "tomli~=2.0.2",
     "tomli-w~=1.1.0",

lib/cli/src/crewai_cli/cli.py

@@ -54,6 +54,12 @@ def eval_crew(*args: Any, **kwargs: Any) -> Any:
     return _eval_crew(*args, **kwargs)
 
 
+def eval_models(*args: Any, **kwargs: Any) -> Any:
+    from crewai_cli.experimental.eval_crew import eval_models as _eval_models
+
+    return _eval_models(*args, **kwargs)
+
+
 if TYPE_CHECKING:
     # mypy sees the real classes; at runtime the shims below defer the
     # heavy imports until a command actually instantiates them.
@@ -692,8 +698,52 @@ def run(
         "crewAI recorded for the run."
     ),
 )
-def eval_command(run_id: str | None) -> None:
-    """Evaluate the last traced run through CrewAI AMP."""
+@click.option(
+    "--models",
+    "models",
+    type=str,
+    default=None,
+    metavar="LIST",
+    help=(
+        "Compare models on this project's deployment instead: ONE comma-separated "
+        'list of provider/model, e.g. "openai/gpt-4o-mini,anthropic/claude-haiku-4-5". '
+        "The deployment runs once as deployed and once per model; needs `crewai login`."
+    ),
+)
+@click.option(
+    "--deployment",
+    "deployment_id",
+    type=str,
+    default=None,
+    metavar="UUID",
+    help=(
+        "With --models: the deployment to run, when AMP cannot tell it from the "
+        "project id."
+    ),
+)
+def eval_command(
+    run_id: str | None, models: str | None, deployment_id: str | None
+) -> None:
+    """Evaluate the last traced run through CrewAI AMP, or compare models on the
+    project's deployment (--models).
+
+    A run's evaluation exits 0 only when the goal gate PASSED, and 1 otherwise — a
+    failed gate, no verdict, or an evaluation that could not run — so a CI job can
+    gate on it. A comparison exits 0 when it finished, 1 when it failed or could
+    not start.
+    """
+    if models is not None:
+        if run_id is not None:
+            raise click.UsageError(
+                "--run grades a run that already happened; --models runs the "
+                "deployment again. Give one of them."
+            )
+        eval_models(models, deployment_id=deployment_id)
+        return
+    if deployment_id is not None:
+        raise click.UsageError(
+            "--deployment names the deployment --models runs; add --models LIST."
+        )
     eval_crew(run_id=run_id)
 
 

lib/cli/src/crewai_cli/deploy/main.py

@@ -15,7 +15,11 @@ from crewai_cli.command import BaseCommand, PlusAPIMixin
 from crewai_cli.constants import DEFAULT_CREWAI_ENTERPRISE_URL
 from crewai_cli.deploy.archive import ArchiveError, create_project_zip
 from crewai_cli.deploy.validate import DeployValidator, Severity, render_report
-from crewai_cli.utils import fetch_and_json_env_file, get_project_name
+from crewai_cli.utils import (
+    fetch_and_json_env_file,
+    get_project_id,
+    get_project_name,
+)
 
 
 console = Console()
@@ -232,6 +236,14 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
         BaseCommand.__init__(self)
         PlusAPIMixin.__init__(self, telemetry=self._telemetry)
         self.project_name = get_project_name(require=True)
+        # Sent with every create and push, so AMP knows which project a
+        # deployment runs — `crewai eval --models` finds the deployment by it.
+        # Read, never minted: a deploy does not rewrite pyproject.toml.
+        self.project_id = get_project_id()
+
+    def _project(self) -> dict[str, str]:
+        """`project_id=` for a create or push request, when the project has one."""
+        return {"project_id": self.project_id} if self.project_id else {}
 
     def _standard_no_param_error_message(self) -> None:
         """
@@ -424,10 +436,10 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
             env_vars = fetch_and_json_env_file()
             return self._update_crew_from_zip(deployment_uuid, repository, env_vars)
         if uuid:
-            return self.plus_api_client.deploy_by_uuid(uuid)
+            return self.plus_api_client.deploy_by_uuid(uuid, **self._project())
         if not project_name:
             raise ValueError("project_name is required to deploy by name")
-        return self.plus_api_client.deploy_by_name(project_name)
+        return self.plus_api_client.deploy_by_name(project_name, **self._project())
 
     def _deploy_from_local_source(
         self,
@@ -438,9 +450,9 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
     ) -> Any | None:
         """Deploy using local origin, as before AMP zip_deployment existed."""
         if remote_repo_url and uuid:
-            return self.plus_api_client.deploy_by_uuid(uuid)
+            return self.plus_api_client.deploy_by_uuid(uuid, **self._project())
         if remote_repo_url and project_name:
-            return self.plus_api_client.deploy_by_name(project_name)
+            return self.plus_api_client.deploy_by_name(project_name, **self._project())
         if uuid:
             _display_git_remote_help()
             env_vars = fetch_and_json_env_file()
@@ -627,6 +639,7 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
                 zip_file_path,
                 name=self.project_name,
                 env=env_vars,
+                **self._project(),
             )
         finally:
             zip_file_path.unlink(missing_ok=True)
@@ -649,6 +662,7 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
                 uuid,
                 zip_file_path,
                 env=env_vars,
+                **self._project(),
             )
         finally:
             zip_file_path.unlink(missing_ok=True)
@@ -692,13 +706,16 @@ class DeployCommand(BaseCommand, PlusAPIMixin):
         """
         if not self.project_name:
             raise ValueError("project_name is required to create a deployment payload")
-        return {
+        payload: CreateCrewPayload = {
             "deploy": {
                 "name": self.project_name,
                 "repo_clone_url": remote_repo_url,
                 "env": env_vars,
             }
         }
+        if self.project_id:
+            payload["project_id"] = self.project_id
+        return payload
 
     def _display_creation_success(self, json_response: dict[str, Any]) -> None:
         """

lib/cli/src/crewai_cli/experimental/eval_crew.py

@@ -10,6 +10,13 @@ Who may evaluate what is AMP's decision: an anonymous run once without an
 account, then it needs one; a run traced while logged in for that
 organization's members; a deployment execution for members who may see its
 traces. The command sends the saved `crewai login` when there is one.
+
+`crewai eval --models "a,b"` compares models instead: AMP finds the project's
+deployment by `[tool.crewai].project_id` (or `--deployment`), which runs once as
+deployed and once per model; each run is graded, and the comparison — the four
+grades, cost and time per model, and what would make the agents do better — is
+printed when it is done. It always needs the login: it runs the deployment.
+Every evaluation, of either kind, is filed under the project id.
 """
 
 from __future__ import annotations
@@ -23,8 +30,9 @@ import os
 from pathlib import Path
 import sys
 import time
-from typing import Any
+from typing import Any, NoReturn, cast
 from urllib.parse import urlparse
+import uuid
 import webbrowser
 
 import click
@@ -33,11 +41,16 @@ from crewai_core.settings import Settings
 from dotenv import load_dotenv, set_key
 import httpx
 from rich.console import Console
+from rich.table import Table
 from rich.text import Text
 
 from crewai_cli.authentication.token import AuthError, get_auth_token
 from crewai_cli.plus_api import PlusAPI
-from crewai_cli.utils import get_or_create_project_id, is_dmn_mode_enabled
+from crewai_cli.utils import (
+    get_or_create_project_id,
+    get_project_id,
+    is_dmn_mode_enabled,
+)
 
 
 console = Console()
@@ -107,7 +120,7 @@ def _note(text: str, style: str = "dim") -> None:
 
 def eval_crew(run_id: str | None = None) -> None:
     """Evaluate the last traced run of this project, or the run RUN_ID."""
-    get_or_create_project_id()
+    project_id = get_or_create_project_id()
     # Read before the project's .env is loaded, so a project cannot add itself.
     trusted = _trusted_amp_origins()
     _load_project_env()
@@ -140,6 +153,7 @@ def eval_crew(run_id: str | None = None) -> None:
             client,
             execution_id,
             wait_for_spans=run_id is None and _ran_just_now(record),
+            project_id=project_id,
         )
     except EvaluationStoppedError as stopped:
         _fail(str(stopped))
@@ -168,10 +182,19 @@ def eval_crew(run_id: str | None = None) -> None:
         raise SystemExit(130) from None
     _print_verdict(finished, url)
     _say_where_the_criteria_live(write_eval_config(finished))
-    if finished.get("status") != "done":
+    # The exit code is what a CI job reads, so it is the gate's: 0 only for a
+    # run that PASSED. A failed gate, one without a verdict, and an evaluation
+    # that stopped are all 1 — a pipeline that carried on past any of them would
+    # ship what the evaluation did not vouch for.
+    if not _gate_passed(finished):
         raise SystemExit(1)
 
 
+def _gate_passed(finished: dict[str, Any]) -> bool:
+    verdict = finished.get("verdict") if finished.get("status") == "done" else None
+    return isinstance(verdict, dict) and str(verdict.get("gate")).lower() == "passed"
+
+
 def _ran_just_now(record: dict[str, Any]) -> bool:
     """Did the project record this run in the last few minutes?
 
@@ -325,7 +348,13 @@ def evaluate_run(
     for them is always on here.
     """
     client = _amp_client(_machine_amp_origins(), note=note)
-    started = _start_evaluation(client, execution_id, wait_for_spans=True, note=note)
+    started = _start_evaluation(
+        client,
+        execution_id,
+        wait_for_spans=True,
+        note=note,
+        project_id=get_project_id(),
+    )
     # After the start, exactly as the command counts it: `cli_usage:eval` is the
     # count of evaluations that began, and an evaluation the app runs is one.
     _record_usage(logged_in=client.api_key is not None)
@@ -506,7 +535,9 @@ def _run_and_let_the_app_evaluate() -> str | None:
         f"  1. add {TRACING_ENV_VAR}=true to .env\n  2. crewai run\n  3. crewai eval"
     )
     if is_dmn_mode_enabled() or not sys.stdin.isatty():
-        console.print(steps, style="yellow")
+        # `Text`, never markup: the reason may be an OS error's own words, and
+        # its `[Errno 13]` would be read as a style tag.
+        console.print(Text(_nothing_traced_unattended() or steps), style="yellow")
         raise SystemExit(1)
     if not click.confirm(
         "No traced run is recorded in this project. Turn tracing on and run the crew now? "
@@ -567,6 +598,32 @@ def _recorded_since(record: dict[str, Any], began: datetime) -> bool:
     return when >= began - timedelta(seconds=1)
 
 
+def _nothing_traced_unattended() -> str | None:
+    """Why a run with tracing on left nothing to evaluate, when nobody was there.
+
+    An anonymous run asks before its trace leaves the machine, and a process with
+    no terminal has nobody to ask — so its trace is kept local, tracing on or
+    not. Telling that user to turn tracing on sends them round the same loop;
+    logging in is what makes an unattended run traced. None when tracing is off
+    or there is a login: the ordinary steps are the right ones then. A login
+    that cannot be read says so instead.
+    """
+    if os.environ.get(TRACING_ENV_VAR, "").strip().lower() not in ("true", "1"):
+        return None
+    try:
+        if saved_login() is not None:
+            return None
+    except EvaluationStoppedError as unreadable:
+        # A login that exists and cannot be read is the reason, and its
+        # sentence says what to do about it.
+        return str(unreadable)
+    return (
+        "No traced run is recorded in this project. Tracing is on, but a run nobody is "
+        "watching is only traced when you are logged in: run `crewai login`, then "
+        "`crewai run` and `crewai eval` again."
+    )
+
+
 def _enable_tracing() -> None:
     """`CREWAI_TRACING_ENABLED=true` in the project's .env, and in this process for the run about to start."""
     env_file = Path.cwd() / ".env"
@@ -585,6 +642,7 @@ def _start_evaluation(
     *,
     wait_for_spans: bool = False,
     note: Callable[[str], None] = _note,
+    project_id: str | None = None,
 ) -> dict[str, Any]:
     deadline = time.monotonic() + SPANS_WAIT_SECONDS if wait_for_spans else 0.0
     said = False
@@ -593,7 +651,9 @@ def _start_evaluation(
     eval_config = project_eval_config(note=note)
     while True:
         try:
-            response = client.create_evaluation(execution_id, eval_config=eval_config)
+            response = client.create_evaluation(
+                execution_id, eval_config=eval_config, project_id=project_id
+            )
         except httpx.HTTPError as error:
             raise EvaluationStoppedError(
                 f"Could not reach AMP to start the evaluation: {error}"
@@ -610,6 +670,17 @@ def _start_evaluation(
             time.sleep(SPANS_POLL_SECONDS)
             continue
         break
+    return _accepted(response, f"run {execution_id}", note=note)
+
+
+def _accepted(
+    response: httpx.Response,
+    subject: str,
+    *,
+    note: Callable[[str], None] = _note,
+    about_a_deployment: bool = False,
+) -> dict[str, Any]:
+    """AMP's answer to a start: the evaluation it opened, or its refusal as a sentence."""
     if response.status_code in (200, 202):
         payload = _payload(response)
         # The id is what every later call is made with, so a missing or
@@ -629,7 +700,9 @@ def _start_evaluation(
         raise EvaluationStoppedError(
             f"AMP answered without an evaluation id ({response.status_code})."
         )
-    raise EvaluationStoppedError(_refusal_message(response, f"run {execution_id}"))
+    raise EvaluationStoppedError(
+        _refusal_message(response, subject, about_a_deployment=about_a_deployment)
+    )
 
 
 def _report_url(value: Any) -> str | None:
@@ -668,12 +741,16 @@ def _wait(
     url: str | None,
     *,
     on_status: Callable[[dict[str, Any]], None] | None = None,
+    answer: str = "verdict",
 ) -> dict[str, Any]:
     """Poll until the evaluation is done or failed.
 
     Every unfinished answer goes to ON_STATUS, so a caller that has somewhere to
-    show progress can show it; the caller decides what a Ctrl-C means.
+    show progress can show it; the caller decides what a Ctrl-C means. A done
+    answer must carry a well-formed ANSWER: a run's `verdict`, or a models
+    evaluation's `comparison`.
     """
+    well_formed = _WELL_FORMED[answer]
     where = f" at {url}" if url else ""
     subject = f"evaluation {evaluation_id}"
     misses = (
@@ -701,9 +778,9 @@ def _wait(
         misses = 0
         payload = _payload(response) or {}
         status = payload.get("status")
-        if status == "done" and not _well_formed_verdict(payload.get("verdict")):
+        if status == "done" and not well_formed(payload):
             raise EvaluationStoppedError(
-                f"AMP answered done without a verdict (protocol error); follow it{where or ' on AMP'}."
+                f"AMP answered done without a {answer} (protocol error); follow it{where or ' on AMP'}."
             )
         if status in FINISHED:
             return payload
@@ -716,6 +793,10 @@ def _wait(
         time.sleep(POLL_SECONDS)
 
 
+def _carries_a_verdict(payload: dict[str, Any]) -> bool:
+    return _well_formed_verdict(payload.get("verdict"))
+
+
 def _well_formed_verdict(verdict: Any) -> bool:
     """`{"gate": "<word>", "grades": {area: 1..5 | null}}` — anything else is a
     protocol error. A grade is an exact integer in range: `True` is an `int` to
@@ -732,6 +813,403 @@ def _a_grade(grade: Any) -> bool:
     return grade is None or (type(grade) is int and 1 <= grade <= 5)
 
 
+def _carries_a_comparison(payload: dict[str, Any]) -> bool:
+    """`{"comparison": {"models": [{...}, ...]}}` — the rows are read cell by
+    cell when printed, and a cell that is not what it should be prints as "—";
+    a comparison with no rows to print is a protocol error."""
+    comparison = payload.get("comparison")
+    return (
+        isinstance(comparison, dict)
+        and isinstance(comparison.get("models"), list)
+        and bool(comparison["models"])
+        and all(isinstance(row, dict) for row in comparison["models"])
+    )
+
+
+_WELL_FORMED: dict[str, Callable[[dict[str, Any]], bool]] = {
+    "verdict": _carries_a_verdict,
+    "comparison": _carries_a_comparison,
+}
+
+
+# ── crewai eval --models ─────────────────────────────────────────────────────
+
+MAX_MODELS = 5
+MAX_MODEL_CHARS = 200
+MODELS_EXAMPLE = '--models "openai/gpt-4o-mini,anthropic/claude-haiku-4-5"'
+LOGIN_REQUIRED = (
+    "Comparing models runs your deployment, so it needs your CrewAI AMP account: "
+    "log in with `crewai login` and run it again."
+)
+GRADE_COLUMNS = ("goal", "tasks", "agents", "tools")
+TOP_SUGGESTIONS = 3
+
+
+def parse_models(text: str | None) -> list[str]:
+    """The models to compare, from ONE comma-separated list.
+
+    Items are stripped and a repeat is dropped. Each must name its provider
+    (`provider/model`) — the deployment builds the model from exactly that
+    string, and a bare `gpt-4o` would be a guess about which provider bills it.
+    """
+    models: list[str] = []
+    for raw in (text or "").split(","):
+        item = raw.strip()
+        if not item or item in models:
+            continue
+        provider, separator, name = item.partition("/")
+        if not (separator and provider.strip() and name.strip()):
+            raise EvaluationStoppedError(
+                f"{item!r} names no provider: write each model as provider/model, "
+                f"e.g. {MODELS_EXAMPLE}."
+            )
+        if len(item) > MAX_MODEL_CHARS:
+            raise EvaluationStoppedError(
+                f"{item[:40]!r}… is longer than {MAX_MODEL_CHARS} characters; "
+                "that is not a model name."
+            )
+        models.append(item)
+    if not models:
+        raise EvaluationStoppedError(
+            f"--models names no model: give one to {MAX_MODELS}, e.g. {MODELS_EXAMPLE}."
+        )
+    if len(models) > MAX_MODELS:
+        raise EvaluationStoppedError(
+            f"--models names {len(models)} models; compare at most {MAX_MODELS} at once."
+        )
+    return models
+
+
+def _deployment_id(value: str | None) -> str | None:
+    """`--deployment` as the UUID AMP knows the deployment by, or None."""
+    if value is None:
+        return None
+    try:
+        return str(uuid.UUID(value.strip()))
+    except ValueError:
+        raise EvaluationStoppedError(
+            f"--deployment {value!r} is not a deployment id; it is the UUID AMP "
+            "shows for the deployment."
+        ) from None
+
+
+# Model names that are the same for everyone who types them are public. What
+# else a `--models` item can carry is a customer's: a fine-tune id
+# (`openai/ft:gpt-4o-mini:acme-corp::abc`), an Azure deployment name, a
+# self-hosted model or host. Those are sent as `<provider>/other`.
+OTHER_MODEL = "other"
+# The vendor segment of an aggregator id (`openrouter/openai/gpt-4o-mini`)
+# that is itself public.
+_PUBLIC_VENDORS = frozenset(
+    {"openai", "anthropic", "google", "meta-llama", "mistralai", "deepseek", "qwen"}
+)
+
+
+def telemetry_model_name(model: str) -> str:
+    """MODEL as the usage stats may carry it: as typed when it is a model crewAI
+    knows, else `<provider>/other` — and `other/other` for a provider crewAI does
+    not know, since a provider string can name a host too.
+
+    Known means an exact entry of crewAI's model catalog (the context-window
+    tables every provider resolves against), never a prefix of one: a fine-tune
+    or a deployment named after a public model is still the customer's.
+    """
+    catalog, providers = _known_models_and_providers()
+    provider, _, name = model.partition("/")
+    if provider not in providers:
+        return f"{OTHER_MODEL}/{OTHER_MODEL}"
+    vendor, nested, tail = name.partition("/")
+    public = (
+        name in catalog
+        or model in catalog
+        or (nested and vendor in _PUBLIC_VENDORS and tail in catalog)
+    )
+    if not public or any(part.startswith("ft:") for part in model.split("/")):
+        return f"{provider}/{OTHER_MODEL}"
+    return model
+
+
+def _known_models_and_providers() -> tuple[frozenset[str], frozenset[str]]:
+    """crewAI's catalog of models and the providers it routes; empty when this
+    environment's crewai cannot say, so every model is then sent as "other"."""
+    from crewai_cli.constants import PROVIDERS
+
+    try:
+        from crewai.llm import SUPPORTED_NATIVE_PROVIDERS
+        from crewai.llms.context_window import LLM_CONTEXT_WINDOW_SIZES
+    except Exception:
+        return frozenset(), frozenset()
+    return (
+        frozenset(LLM_CONTEXT_WINDOW_SIZES),
+        frozenset(SUPPORTED_NATIVE_PROVIDERS) | frozenset(PROVIDERS),
+    )
+
+
+def _record_models_usage(models: list[str]) -> None:
+    """Count a comparison that is actually starting, and which models it compares.
+
+    The models go through `telemetry_model_name`: a public model by name, any
+    other as `<provider>/other`. Nothing names the run, the deployment or the
+    organization. Always logged in: a comparison cannot start without the
+    account.
+    """
+    try:
+        from crewai_core.telemetry import Telemetry
+
+        telemetry = Telemetry()
+        telemetry.set_tracer()
+        telemetry.feature_usage_span(
+            "cli_usage:eval_models",
+            {
+                "authenticated": "true",
+                "models": ",".join(telemetry_model_name(m) for m in models),
+                "models_count": str(len(models)),
+            },
+        )
+    except Exception:  # noqa: S110 - telemetry must never break a command
+        pass
+
+
+def eval_models(models_text: str, deployment_id: str | None = None) -> None:
+    """Run this project's deployment once as deployed and once per model, and compare.
+
+    The deployment is AMP's to find, by the project id, unless DEPLOYMENT_ID
+    names one; its own models are the baseline, read off the deployment rather
+    than this checkout, which may differ from what was deployed.
+    """
+    try:
+        models = parse_models(models_text)
+        deployment = _deployment_id(deployment_id)
+    except EvaluationStoppedError as stopped:
+        _fail(str(stopped))
+    if not Path("pyproject.toml").is_file():
+        _fail(
+            "No crewAI project here (no pyproject.toml). Run `crewai eval --models` "
+            "from the directory of the project you deployed."
+        )
+    project_id = get_or_create_project_id()
+    if not project_id:
+        _fail(
+            "Could not read or write [tool.crewai].project_id in pyproject.toml, which "
+            "is how AMP finds this project's deployment."
+        )
+    # Read before the project's .env is loaded, so a project cannot add itself.
+    trusted = _trusted_amp_origins()
+    _load_project_env()
+    try:
+        client = _amp_client(trusted)
+    except EvaluationStoppedError as stopped:
+        _fail(str(stopped))
+    if client.api_key is None:
+        _fail(LOGIN_REQUIRED)
+
+    try:
+        response = client.create_models_evaluation(
+            models,
+            project_id=project_id,
+            eval_config=project_eval_config(),
+            deployment_id=deployment,
+        )
+    except httpx.HTTPError as error:
+        _fail(f"Could not reach AMP to start the comparison: {error}")
+    try:
+        started = _accepted(response, "the comparison", about_a_deployment=True)
+    except EvaluationStoppedError as stopped:
+        _fail(str(stopped))
+    # After, not before, as `cli_usage:eval` is counted: a refused request —
+    # no deployment, one this account may not run — is not a comparison.
+    _record_models_usage(models)
+    url = started.get("url")
+    console.print(
+        Text("Comparing ")
+        .append(", ".join(models), style="bold")
+        .append(" with the deployed models")
+    )
+    if url:
+        # Appended, never interpolated — the same reason as `eval_crew`'s link.
+        console.print(Text("Follow it at ").append(url, style="cyan underline"))
+        _open(url)
+
+    console.print("Waiting for the comparison…", style="dim")
+    shown: list[str] = []
+
+    def show_progress(payload: dict[str, Any]) -> None:
+        line = _progress_line(payload)
+        if line and (not shown or shown[-1] != line):
+            shown.append(line)
+            _note(line)
+
+    try:
+        finished = _wait(
+            client, started["id"], url, on_status=show_progress, answer="comparison"
+        )
+    except EvaluationStoppedError as stopped:
+        _fail(str(stopped))
+    except KeyboardInterrupt:
+        console.print(
+            Text(f"\nStill running{f' at {url}' if url else ''}."), style="yellow"
+        )
+        raise SystemExit(130) from None
+    _print_comparison(finished, url)
+    # The criteria the comparison was graded on, for the project to edit — only
+    # when it has none, exactly as after a Mode 1 evaluation.
+    _say_where_the_criteria_live(write_eval_config(finished))
+    if finished.get("status") != "done":
+        raise SystemExit(1)
+
+
+def _progress_line(payload: dict[str, Any]) -> str | None:
+    """One line for where the comparison is, from the poll's `progress` (or the
+    last of its `events`): which model is running, "model 2 of 3", and the
+    subject being judged. What is not there is left out, never guessed."""
+    progress = payload.get("progress")
+    events = payload.get("events")
+    if progress is None and isinstance(events, list) and events:
+        progress = events[-1]
+    if isinstance(progress, str):
+        return progress.strip() or None
+    if not isinstance(progress, dict):
+        return None
+    if isinstance(progress.get("message"), str) and progress["message"].strip():
+        return str(progress["message"]).strip()
+    event = progress.get("event")
+    detail = progress.get("payload")
+    if not isinstance(detail, dict):
+        detail = progress
+    index, total = detail.get("index"), detail.get("total")
+    position = (
+        f"model {index + 1} of {total}"
+        if type(index) is int and type(total) is int and 0 <= index < total
+        else None
+    )
+    name = next(
+        (
+            str(detail[key])
+            for key in ("label", "key")
+            if isinstance(detail.get(key), str) and detail[key]
+        ),
+        None,
+    )
+    subject = detail.get("subject")
+    if event == "judging" or isinstance(subject, str):
+        parts = [f"judging {subject}" if isinstance(subject, str) else "judging"]
+    elif event == "configuration_done":
+        parts = ["graded"]
+    else:
+        parts = ["running"]
+    parts += [part for part in (position, name) if part]
+    known = event is not None or isinstance(subject, str) or len(parts) > 1
+    return " · ".join(parts) if known else None
+
+
+def _print_comparison(finished: dict[str, Any], url: str | None) -> None:
+    """The models side by side, then what would make them better.
+
+    Every cell came over the wire, so each is a `Text`: a label such as
+    `Writer: [red]x[/red]` prints as written. The baseline — the deployment as
+    it is — is marked, and the best value in each column is starred: the highest
+    grade, the lowest cost and time. A column where every model is the same, or
+    only one has a value, stars nothing: there is no "best" to point at.
+    """
+    if finished.get("status") != "done":
+        console.print(
+            Text(f"Comparison failed: {finished.get('error') or 'no reason given'}"),
+            style="bold red",
+        )
+        if url:
+            console.print(Text(f"Report: {url}"))
+        return
+    comparison = finished["comparison"]  # _wait let only a well-formed one through
+    rows: list[dict[str, Any]] = comparison["models"]
+
+    grades = {area: [_grade_of(row, area) for row in rows] for area in GRADE_COLUMNS}
+    costs = [_number(row.get("cost_usd")) for row in rows]
+    seconds = [_number(row.get("seconds")) for row in rows]
+    best = {area: _best(values, max) for area, values in grades.items()}
+    cheapest, fastest = _best(costs, min), _best(seconds, min)
+
+    table = Table(show_edge=False, pad_edge=False)
+    for header in ("model", *GRADE_COLUMNS, "cost", "time"):
+        table.add_column(header, justify="left" if header == "model" else "right")
+    for n, row in enumerate(rows):
+        label = Text(str(row.get("label") or row.get("key") or "?"))
+        if row.get("baseline") is True:
+            label.append(" (deployed)", style="dim")
+        cells = [label]
+        for area in GRADE_COLUMNS:
+            grade = grades[area][n]
+            cells.append(
+                _starred(f"{grade}/5" if grade is not None else "—", grade, best[area])
+            )
+        cost, took = costs[n], seconds[n]
+        cells.append(
+            _starred(f"${cost:.4f}" if cost is not None else "—", cost, cheapest)
+        )
+        cells.append(
+            _starred(f"{took:.1f}s" if took is not None else "—", took, fastest)
+        )
+        table.add_row(*cells)
+    console.print(table)
+
+    suggestions = _top_suggestions(comparison.get("suggestions"))
+    if suggestions:
+        console.print(Text("What would make it better", style="bold"))
+    for n, item in enumerate(suggestions, 1):
+        where = " — ".join(
+            str(item[key])
+            for key in ("subject", "field")
+            if isinstance(item.get(key), str)
+        )
+        line = Text(f"{n}. ")
+        if where:
+            line.append(where, style="bold").append(": ")
+        line.append(str(item.get("problem") or ""))
+        if item.get("shared") is True:
+            line.append(" (every model)", style="dim")
+        console.print(line)
+        if isinstance(item.get("change"), str) and item["change"].strip():
+            console.print(Text(f"   change: {item['change'].strip()}"))
+    if url:
+        console.print(Text(f"Full report: {url}"))
+
+
+def _grade_of(row: dict[str, Any], area: str) -> int | None:
+    grades = row.get("grades")
+    grade = grades.get(area) if isinstance(grades, dict) else None
+    return grade if grade is not None and _a_grade(grade) else None
+
+
+def _number(value: Any) -> float | None:
+    """A non-negative number, or None: `True` is an `int` to Python and not a cost."""
+    if isinstance(value, bool) or not isinstance(value, (int, float)):
+        return None
+    return float(value) if value >= 0 else None
+
+
+def _best(values: list[Any], pick: Callable[[list[Any]], Any]) -> float | int | None:
+    known = [value for value in values if value is not None]
+    if len(known) < 2 or len(set(known)) == 1:
+        return None
+    return cast(float | int, pick(known))
+
+
+def _starred(text: str, value: Any, best: Any) -> Text:
+    cell = Text(text)
+    if best is not None and value == best:
+        cell.append(" ★", style="yellow")
+    return cell
+
+
+def _top_suggestions(value: Any) -> list[dict[str, Any]]:
+    """The first few suggestions, the ones every model needed first — a problem
+    every model had points at the prompt, not at a model."""
+    if not isinstance(value, list):
+        return []
+    items = [item for item in value if isinstance(item, dict) and item.get("problem")]
+    items.sort(key=lambda item: item.get("shared") is not True)
+    return items[:TOP_SUGGESTIONS]
+
+
 def _print_verdict(finished: dict[str, Any], url: str | None) -> None:
     """Everything printed here came over the wire, so it is composed as `Text`
     and never as markup: a `Console` parses square brackets, and an area named
@@ -780,15 +1258,26 @@ def _payload(response: httpx.Response) -> dict[str, Any] | None:
     return loaded if isinstance(loaded, dict) else None
 
 
-def _refusal_message(response: httpx.Response, subject: str) -> str:
+def _refusal_message(
+    response: httpx.Response, subject: str, *, about_a_deployment: bool = False
+) -> str:
     """AMP's own words when it sent them. SUBJECT is "run <id>" or "evaluation <id>".
 
     A sentence, not a print: the terminal and the run app both show it, and only
-    one of them shows it by printing.
+    one of them shows it by printing. ABOUT_A_DEPLOYMENT: a 403 that is not about
+    the credential — this account may not run that deployment — is AMP's
+    sentence alone, because logging in again changes nothing there.
     """
     payload = _payload(response) or {}
     message = str(payload.get("message") or "").strip()
     error = str(payload.get("error") or "")
+    if (
+        about_a_deployment
+        and response.status_code == 403
+        and error not in {"bad_credentials", "account_required"}
+        and message
+    ):
+        return message
     if response.status_code in (401, 403):
         if error == "account_required" and message:
             return message
@@ -806,7 +1295,7 @@ def _refused(response: httpx.Response, subject: str) -> None:
     _fail(_refusal_message(response, subject))
 
 
-def _fail(message: str) -> None:
+def _fail(message: str) -> NoReturn:
     # `Text`, because most of what reaches here is AMP's own sentence and a
     # `Console` parses square brackets. No caller relies on markup; the colour
     # comes from `style`.

lib/cli/src/crewai_cli/plus_api.py

@@ -28,17 +28,56 @@ class PlusAPI(_CorePlusAPI):
     EVALUATION_POLL_TIMEOUT = 30.0
 
     def create_evaluation(
-        self, execution_id: str, *, eval_config: str | None = None
+        self,
+        execution_id: str,
+        *,
+        eval_config: str | None = None,
+        project_id: str | None = None,
     ) -> httpx.Response:
         """Ask AMP to evaluate the traced run EXECUTION_ID (crewai eval).
 
         EVAL_CONFIG is the project's own `eval.jsonc` when it has one: what
         good means for this crew, in its own words. Sent as it was written,
-        comments and all, and read by the grader rather than here.
+        comments and all, and read by the grader rather than here. PROJECT_ID
+        is `[tool.crewai].project_id`, so the evaluation is filed under the
+        project it is about.
         """
         body: dict[str, str] = {"execution_id": execution_id}
         if eval_config:
             body["eval_config"] = eval_config
+        if project_id:
+            body["project_id"] = project_id
+        return self._make_request(
+            "POST",
+            self.EVALUATIONS_RESOURCE,
+            json=body,
+            timeout=self.EVALUATION_START_TIMEOUT,
+        )
+
+    def create_models_evaluation(
+        self,
+        models: list[str],
+        *,
+        project_id: str,
+        eval_config: str | None = None,
+        deployment_id: str | None = None,
+    ) -> httpx.Response:
+        """Ask AMP to run the project's deployment once per model and compare them.
+
+        AMP finds the deployment by PROJECT_ID (`[tool.crewai].project_id`)
+        unless DEPLOYMENT_ID names one. MODELS are `provider/model` strings;
+        the deployment's own models are the baseline, so they are not sent.
+        EVAL_CONFIG is the project's `eval.jsonc`, as `create_evaluation` sends it.
+        """
+        body: dict[str, Any] = {
+            "kind": "models",
+            "project_id": project_id,
+            "models": list(models),
+        }
+        if eval_config:
+            body["eval_config"] = eval_config
+        if deployment_id:
+            body["deployment_id"] = deployment_id
         return self._make_request(
             "POST",
             self.EVALUATIONS_RESOURCE,
@@ -90,11 +129,18 @@ class PlusAPI(_CorePlusAPI):
         *,
         name: str | None = None,
         env: dict[str, str] | None = None,
+        project_id: str | None = None,
     ) -> httpx.Response:
-        """Create a crew deployment from a local project ZIP archive."""
+        """Create a crew deployment from a local project ZIP archive.
+
+        PROJECT_ID is `[tool.crewai].project_id`, so AMP knows which project the
+        deployment runs; sent only when the project has one.
+        """
         data: dict[str, str] = {}
         if name:
             data["name"] = name
+        if project_id:
+            data["project_id"] = project_id
         if env:
             data.update({f"env[{key}]": value for key, value in env.items()})
         return self._make_multipart_request(
@@ -111,9 +157,15 @@ class PlusAPI(_CorePlusAPI):
         zip_file_path: str | Path,
         *,
         env: dict[str, str] | None = None,
+        project_id: str | None = None,
     ) -> httpx.Response:
-        """Update an existing crew deployment from a local project ZIP archive."""
+        """Update an existing crew deployment from a local project ZIP archive.
+
+        PROJECT_ID as for `create_crew_from_zip`.
+        """
         data: dict[str, str] = {}
+        if project_id:
+            data["project_id"] = project_id
         if env:
             data.update({f"env[{key}]": value for key, value in env.items()})
         return self._make_multipart_request(

lib/cli/tests/deploy/test_deploy_main.py

@@ -1316,3 +1316,128 @@ class TestDeployCommand(unittest.TestCase):
         from crewai_cli.version import get_crewai_version
 
         assert isinstance(get_crewai_version(), str)
+
+
+PROJECT_ID = "1f0c6a3e-8d0b-4c4e-9a52-6a9f6d1d2b7e"
+
+
+class TestDeploySendsTheProjectId(unittest.TestCase):
+    """Every create and push carries `[tool.crewai].project_id` when the
+    project has one, so AMP knows which project a deployment runs."""
+
+    @patch("crewai_cli.command.get_auth_token", return_value="test_token")
+    @patch("crewai_cli.deploy.main.get_project_id", return_value=PROJECT_ID)
+    @patch("crewai_cli.deploy.main.get_project_name", return_value="test_project")
+    @patch("crewai_cli.command.PlusAPI")
+    def setUp(self, _plus_api, _name, _project_id, _token):
+        self.deploy_command = deploy_main.DeployCommand()
+        self.mock_client = self.deploy_command.plus_api_client
+        self.mock_client.base_url = "https://app.crewai.com"
+        browser = patch("crewai_cli.deploy.main.webbrowser.open")
+        browser.start()
+        self.addCleanup(browser.stop)
+
+    def _created(self, uuid: str = "new-uuid") -> MagicMock:
+        response = MagicMock()
+        response.status_code = 201
+        response.is_success = True
+        response.json.return_value = {"uuid": uuid, "status": "created"}
+        return response
+
+    def _status(self, zip_deployment: bool) -> MagicMock:
+        response = MagicMock()
+        response.is_success = True
+        response.json.return_value = {"uuid": "test-uuid", "zip_deployment": zip_deployment}
+        return response
+
+    def test_the_command_reads_the_project_id(self):
+        self.assertEqual(self.deploy_command.project_id, PROJECT_ID)
+
+    @patch("crewai_cli.deploy.main.fetch_and_json_env_file", return_value={})
+    @patch("crewai_cli.deploy.main.git.Repository")
+    def test_create_from_git_sends_it(self, mock_repository, _env):
+        mock_repository.return_value.origin_url.return_value = "https://github.com/t/r.git"
+        mock_repository.return_value.create_initial_commit_if_needed.return_value = False
+        self.mock_client.create_crew.return_value = self._created()
+
+        with patch("sys.stdout", new=StringIO()):
+            self.deploy_command.create_crew(confirm=True, skip_validate=True)
+
+        payload = self.mock_client.create_crew.call_args.args[0]
+        # Once, at the top level, where AMP reads it on every deploy route.
+        self.assertEqual(payload["project_id"], PROJECT_ID)
+        self.assertNotIn("project_id", payload["deploy"])
+
+    @patch("crewai_cli.deploy.main.create_project_zip", return_value=Path("/tmp/p.zip"))
+    @patch("crewai_cli.deploy.main.fetch_and_json_env_file", return_value={})
+    @patch("crewai_cli.deploy.main.git.Repository")
+    def test_create_from_a_zip_sends_it(self, mock_repository, _env, _zip):
+        mock_repository.return_value.origin_url.return_value = None
+        mock_repository.return_value.create_initial_commit_if_needed.return_value = False
+        self.mock_client.create_crew_from_zip.return_value = self._created()
+
+        with patch("sys.stdout", new=StringIO()):
+            self.deploy_command.create_crew(confirm=True, skip_validate=True)
+
+        self.mock_client.create_crew_from_zip.assert_called_once_with(
+            Path("/tmp/p.zip"), name="test_project", env={}, project_id=PROJECT_ID
+        )
+
+    @patch("crewai_cli.deploy.main.git.Repository")
+    @patch("crewai_cli.deploy.main.DeployCommand._display_deployment_info")
+    def test_a_push_from_git_sends_it(self, _display, mock_repository):
+        mock_repository.return_value.origin_url.return_value = "https://github.com/t/r.git"
+        mock_repository.return_value.create_initial_commit_if_needed.return_value = False
+        self.mock_client.crew_status_by_uuid.return_value = self._status(False)
+        self.mock_client.crew_status_by_name.return_value = self._status(False)
+
+        self.deploy_command.deploy(uuid="test-uuid", skip_validate=True)
+        self.deploy_command.deploy(skip_validate=True)
+
+        self.mock_client.deploy_by_uuid.assert_called_once_with(
+            "test-uuid", project_id=PROJECT_ID
+        )
+        self.mock_client.deploy_by_name.assert_called_once_with(
+            "test_project", project_id=PROJECT_ID
+        )
+
+    @patch("crewai_cli.deploy.main.create_project_zip", return_value=Path("/tmp/p.zip"))
+    @patch("crewai_cli.deploy.main.fetch_and_json_env_file", return_value={})
+    @patch("crewai_cli.deploy.main.git.Repository")
+    @patch("crewai_cli.deploy.main.DeployCommand._display_deployment_info")
+    def test_a_push_from_a_zip_sends_it(self, _display, mock_repository, _env, _zip):
+        mock_repository.return_value.origin_url.return_value = None
+        mock_repository.return_value.create_initial_commit_if_needed.return_value = False
+        self.mock_client.crew_status_by_uuid.return_value = self._status(True)
+
+        self.deploy_command.deploy(uuid="test-uuid", skip_validate=True)
+
+        self.mock_client.update_crew_from_zip.assert_called_once_with(
+            "test-uuid", Path("/tmp/p.zip"), env={}, project_id=PROJECT_ID
+        )
+
+
+def test_a_project_without_an_id_sends_none(monkeypatch, tmp_path: Path):
+    """Read, never minted: a deploy does not rewrite pyproject.toml."""
+    monkeypatch.chdir(tmp_path)
+    (tmp_path / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+    monkeypatch.setattr("crewai_cli.command.get_auth_token", lambda: "t")
+    monkeypatch.setattr("crewai_cli.command.PlusAPI", MagicMock())
+    monkeypatch.setattr(deploy_main, "get_project_name", lambda require=False: "demo")
+
+    command = deploy_main.DeployCommand()
+
+    assert command.project_id is None and command._project() == {}
+    assert "project_id" not in (tmp_path / "pyproject.toml").read_text()
+
+
+def test_the_project_id_is_read_from_pyproject(monkeypatch, tmp_path: Path):
+    monkeypatch.chdir(tmp_path)
+    (tmp_path / "pyproject.toml").write_text(
+        f"[project]\nname = 'demo'\n\n[tool.crewai]\nproject_id = \"{PROJECT_ID}\"\n"
+    )
+    monkeypatch.setattr("crewai_cli.command.get_auth_token", lambda: "t")
+    monkeypatch.setattr("crewai_cli.command.PlusAPI", MagicMock())
+    monkeypatch.setattr(deploy_main, "get_project_name", lambda require=False: "demo")
+
+    assert deploy_main.DeployCommand()._project() == {"project_id": PROJECT_ID}

lib/cli/tests/experimental/test_eval_crew.py

@@ -44,9 +44,10 @@ class FakeAMP:
         self.api_key = None
         self.headers: dict[str, str] = {"X-Crewai-Organization-Id": "org-42"}
 
-    def create_evaluation(self, execution_id, *, eval_config=None):
+    def create_evaluation(self, execution_id, *, eval_config=None, project_id=None):
         self.calls.append(("create", execution_id))
         self.sent_config = eval_config
+        self.sent_project_id = project_id
         if isinstance(self.create, list):
             return self.create.pop(0) if len(self.create) > 1 else self.create[0]
         return self.create
@@ -241,13 +242,33 @@ def test_run_names_another_execution_and_an_anonymous_caller_sends_no_token(proj
     monkeypatch.setattr(eval_module, "saved_login", lambda: None)
     amp = install(monkeypatch, FakeAMP(statuses=[done("failed")]))
 
-    eval_module.eval_crew(run_id="other-run")
+    with pytest.raises(SystemExit):  # a failed gate is exit 1
+        eval_module.eval_crew(run_id="other-run")
 
     assert amp.api_key is None
     assert amp.calls[0] == ("create", "other-run")
     assert "Goal gate: FAILED" in capsys.readouterr().out
 
 
+@pytest.mark.parametrize(
+    "gate, code", [("passed", None), ("failed", 1), ("inconclusive", 1), ("unknown", 1)]
+)
+def test_the_exit_code_is_the_gates(project, monkeypatch, capsys, gate, code):
+    """A CI job reads the exit code. Only a gate that PASSED is 0: a failed one,
+    and one with no verdict, must stop the pipeline rather than wave it on."""
+    directory, _ = project
+    record_last_run(directory)
+    install(monkeypatch, FakeAMP(statuses=[done(gate)]))
+
+    if code is None:
+        eval_module.eval_crew()
+    else:
+        with pytest.raises(SystemExit) as exited:
+            eval_module.eval_crew()
+        assert exited.value.code == code
+    assert f"Goal gate: {gate.upper()}" in capsys.readouterr().out
+
+
 def test_a_failed_evaluation_exits_one_with_amps_reason(project, monkeypatch, capsys):
     directory, _ = project
     record_last_run(directory)
@@ -659,7 +680,7 @@ def test_an_oversized_config_is_said_once_and_through_the_callers_note(
     answers = iter([httpx.Response(404, json={}), httpx.Response(404, json={}), QUEUED])
 
     class Client:
-        def create_evaluation(self, execution_id, *, eval_config=None):
+        def create_evaluation(self, execution_id, *, eval_config=None, project_id=None):
             sent.append(eval_config)
             return next(answers)
 
@@ -724,7 +745,7 @@ def test_only_a_web_address_is_printed_and_opened(url, opened):
 
 def test_a_report_url_that_is_not_a_web_address_is_dropped_with_a_note(monkeypatch):
     class Client:
-        def create_evaluation(self, execution_id, *, eval_config=None):
+        def create_evaluation(self, execution_id, *, eval_config=None, project_id=None):
             return httpx.Response(
                 202, json={"id": "ev-1", "url": "file:///etc/passwd", "status": "queued"}
             )
@@ -1126,3 +1147,456 @@ def test_read_last_run_reads_the_record_crewai_writes(tmp_path):
     record_last_run(tmp_path)
     record = eval_module.read_last_run(tmp_path)
     assert record is not None and record["execution_id"] == EXECUTION_ID and record["amp_base_url"] == "https://amp.test"
+
+
+@pytest.mark.parametrize(
+    "tracing, login, says_login",
+    [
+        ("true", None, True),  # tracing on, nobody logged in: log in
+        ("true", "tok", False),  # logged in: the ordinary steps
+        (None, None, False),  # tracing off: turn it on
+    ],
+)
+def test_an_unattended_run_with_nothing_traced_says_what_would_trace_it(
+    project, monkeypatch, capsys, tracing, login, says_login
+):
+    """With no terminal, an anonymous run's trace stays on the machine even with
+    tracing on. Telling that user to turn tracing on sends them round the same
+    loop; logging in is what makes an unattended run traced."""
+    directory, _ = project
+    (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+    monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+    monkeypatch.setattr(eval_module, "saved_login", lambda: login)
+    if tracing:
+        monkeypatch.setenv("CREWAI_TRACING_ENABLED", tracing)
+    else:
+        monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+
+    with pytest.raises(SystemExit) as exited:
+        eval_module.eval_crew()
+
+    out = capsys.readouterr().out.replace("\n", " ")
+    assert exited.value.code == 1
+    assert ("run `crewai login`" in out) is says_login
+    assert ("add CREWAI_TRACING_ENABLED=true" in out) is not says_login
+
+
+def test_an_unreadable_login_is_the_reason_given_when_nothing_was_traced(
+    project, monkeypatch, capsys
+):
+    """A login that exists and cannot be read is why an unattended run was not
+    traced, and its own sentence says what to do — not "turn tracing on"."""
+    directory, _ = project
+    (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+    monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+    monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
+
+    def unreadable() -> None:
+        raise eval_module.EvaluationStoppedError(
+            "Could not read the saved login (PermissionError: [Errno 13] "
+            "Permission denied: [/Users/me/.config/crewai]). Run `crewai login` again"
+        )
+
+    monkeypatch.setattr(eval_module, "saved_login", unreadable)
+
+    with pytest.raises(SystemExit):
+        eval_module.eval_crew()
+
+    out = capsys.readouterr().out.replace("\n", " ")
+    assert "Could not read the saved login" in out
+    # printed as it is: `[/Users/…]` read as markup is a closing tag, and a crash
+    assert "[/Users/me/.config/crewai]" in out
+    assert "add CREWAI_TRACING_ENABLED=true" not in out
+
+
+# ── Mode 1 files the evaluation under the project ───────────────────────────
+
+
+def test_the_evaluation_is_filed_under_the_project(project, monkeypatch):
+    directory, _ = project
+    record_last_run(directory)
+    monkeypatch.setattr(eval_module, "get_or_create_project_id", lambda: "proj-1")
+    amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+    eval_module.eval_crew()
+
+    assert amp.sent_project_id == "proj-1"
+
+
+def test_the_app_files_its_evaluation_under_the_project_too(project, monkeypatch):
+    """The run app's door reads the id without minting one: it runs inside a
+    crew's process, where rewriting pyproject.toml is not its business."""
+    monkeypatch.setattr(eval_module, "get_project_id", lambda: "proj-2")
+    amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+    eval_module.evaluate_run(EXECUTION_ID, on_started=lambda started: None)
+
+    assert amp.sent_project_id == "proj-2"
+
+
+# ── crewai eval --models ─────────────────────────────────────────────────────
+
+MODELS = "openai/gpt-4o-mini, openrouter/meta-llama/llama-4-maverick"
+DEPLOYMENT = "9b0a4a53-3f7c-4a8e-9a55-2f1c1e0f8e11"
+
+
+def comparison(**overrides):
+    body = {
+        "models": [
+            {
+                "key": "base", "label": "Poem composer: openai/gpt-5.6-sol", "baseline": True,
+                "grades": {"goal": 5, "tasks": 4, "agents": 3, "tools": None},
+                "gate": "passed", "cost_usd": 0.0040, "seconds": 9.5, "tokens": 900,
+            },
+            {
+                "key": "mini", "label": "Poem composer: openai/gpt-4o-mini", "baseline": False,
+                "grades": {"goal": 5, "tasks": 5, "agents": 2, "tools": None},
+                "gate": "passed", "cost_usd": 0.0012, "seconds": 7.6, "tokens": 736,
+            },
+        ],
+        "suggestions": [
+            {"subject": "agent:Poem composer", "field": "backstory", "problem": "only one model had it",
+             "evidence": "x", "change": "only-one change", "models_affected": ["mini"], "shared": False},
+            {"subject": "agent:Poem composer", "field": "goal", "problem": "the poem ignores the topic",
+             "evidence": "y", "change": "Write a poem about {topic}.", "models_affected": ["base", "mini"],
+             "shared": True},
+            {"subject": "task:write", "field": "expected_output", "problem": "no length", "change": "4 lines"},
+            {"subject": "agent:Poem composer", "field": "tools", "problem": "a fourth", "change": "never shown"},
+        ],
+    }
+    body.update(overrides)
+    return body
+
+
+def compared(**overrides):
+    return httpx.Response(200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": comparison(**overrides)})
+
+
+class FakeModelsAMP(FakeAMP):
+    def __init__(self, create=None, statuses=None):
+        super().__init__(create=create if create is not None else httpx.Response(
+            202, json={"id": "ev-9", "url": URL, "status": "queued"}
+        ), statuses=statuses)
+        self.sent: dict | None = None
+
+    def create_models_evaluation(self, models, *, project_id, eval_config=None, deployment_id=None):
+        self.calls.append(("create_models", tuple(models)))
+        self.sent = {"models": models, "project_id": project_id, "eval_config": eval_config,
+                     "deployment_id": deployment_id}
+        return self.create
+
+
+@pytest.fixture
+def deployed(project, monkeypatch):
+    directory, opened = project
+    (directory / "pyproject.toml").write_text('[tool.crewai]\nproject_id = "proj-1"\n')
+    monkeypatch.setattr(eval_module, "get_or_create_project_id", lambda: "proj-1")
+    return directory, opened
+
+
+def test_models_are_compared_on_the_deployment_and_the_table_printed(deployed, monkeypatch, capsys):
+    directory, opened = deployed
+    (directory / "eval.jsonc").write_text('{"dataset": []}')
+    running = httpx.Response(200, json={"id": "ev-9", "status": "running", "progress": {
+        "event": "configuration_started", "payload": {"index": 1, "total": 3, "key": "mini"}}})
+    judging = httpx.Response(200, json={"id": "ev-9", "status": "running", "progress": {
+        "event": "judging", "payload": {"subject": "agent 'Poem composer'", "index": 1, "total": 3}}})
+    amp = install(monkeypatch, FakeModelsAMP(statuses=[running, running, judging, compared()]))
+
+    eval_module.eval_models(MODELS)
+
+    out = capsys.readouterr().out
+    assert amp.api_key == "login-token"
+    assert amp.sent == {
+        "models": ["openai/gpt-4o-mini", "openrouter/meta-llama/llama-4-maverick"],
+        "project_id": "proj-1", "eval_config": '{"dataset": []}', "deployment_id": None,
+    }
+    assert opened == [URL] and URL in out
+    # Progress: said once per change, never once per poll.
+    assert out.count("running · model 2 of 3 · mini") == 1
+    assert "judging agent 'Poem composer' · model 2 of 3" in out
+    lines = out.splitlines()
+    base = next(line for line in lines if "gpt-5.6-sol" in line)
+    mini = next(line for line in lines if "Poem composer: openai/gpt-4o-mini" in line)
+    assert "(deployed)" in base and "(deployed)" not in mini
+    # The best per column: tasks and agents differ, goal is a tie, tools nothing.
+    assert "5/5 ★" in mini and "4/5" in base and "4/5 ★" not in base
+    assert "3/5 ★" in base and "2/5 ★" not in mini
+    assert "$0.0012 ★" in mini and "7.6s ★" in mini and "9.5s ★" not in base
+    assert "5/5 ★" not in base  # goal: both 5, nothing to point at
+    # The top three suggestions, the one every model needed first.
+    assert "What would make it better" in out
+    assert out.index("the poem ignores the topic") < out.index("only one model had it")
+    assert "change: Write a poem about {topic}." in out
+    assert "a fourth" not in out
+
+
+def test_the_comparison_is_counted_with_the_models_and_nothing_that_names_it(deployed, monkeypatch):
+    spans: list[tuple[str, dict[str, str]]] = []
+
+    class FakeTelemetry:
+        def set_tracer(self) -> None:
+            pass
+
+        def feature_usage_span(self, feature, attributes=None) -> None:
+            spans.append((feature, attributes or {}))
+
+    monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
+    install(monkeypatch, FakeModelsAMP(statuses=[compared()]))
+
+    eval_module.eval_models(MODELS)
+
+    assert spans == [("cli_usage:eval_models", {
+        "authenticated": "true",
+        # A public model by name; one crewAI's catalog does not list, by provider.
+        "models": "openai/gpt-4o-mini,openrouter/other",
+        "models_count": "2",
+    })]
+
+
+def test_a_refused_comparison_is_not_counted(deployed, monkeypatch, capsys):
+    spans: list[str] = []
+
+    class FakeTelemetry:
+        def set_tracer(self) -> None:
+            pass
+
+        def feature_usage_span(self, feature, attributes=None) -> None:
+            spans.append(feature)
+
+    monkeypatch.setattr("crewai_core.telemetry.Telemetry", FakeTelemetry)
+    refused = httpx.Response(404, json={"error": "deployment_not_found", "message": (
+        "No deployment of this project; deploy it with `crewai deploy create`, or pass --deployment.")})
+    install(monkeypatch, FakeModelsAMP(create=refused))
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models(MODELS)
+
+    assert stopped.value.code == 1 and spans == []
+    assert "deploy it with `crewai deploy create`, or pass --deployment." in capsys.readouterr().out
+
+
+def test_the_deployment_named_is_sent(deployed, monkeypatch):
+    amp = install(monkeypatch, FakeModelsAMP(statuses=[compared()]))
+
+    eval_module.eval_models("openai/gpt-4o-mini", deployment_id=f" {DEPLOYMENT.upper()} ")
+
+    assert amp.sent["deployment_id"] == DEPLOYMENT
+
+
+def test_a_deployment_that_is_not_a_uuid_is_refused_before_anything_is_sent(deployed, monkeypatch, capsys):
+    amp = install(monkeypatch, FakeModelsAMP())
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models("openai/gpt-4o-mini", deployment_id="poem_creator")
+
+    assert stopped.value.code == 1 and amp.calls == []
+    assert "is not a deployment id" in capsys.readouterr().out
+
+
+def test_comparing_models_needs_the_login(deployed, monkeypatch, capsys):
+    monkeypatch.setattr(eval_module, "saved_login", lambda: None)
+    amp = install(monkeypatch, FakeModelsAMP())
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models(MODELS)
+
+    assert stopped.value.code == 1 and amp.calls == []
+    assert "log in with `crewai login`" in capsys.readouterr().out
+
+
+def test_comparing_models_happens_from_the_project(project, monkeypatch, capsys):
+    amp = install(monkeypatch, FakeModelsAMP())
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models(MODELS)
+
+    assert stopped.value.code == 1 and amp.calls == []
+    assert "No crewAI project here" in capsys.readouterr().out
+
+
+@pytest.mark.parametrize(
+    ("text", "expected"),
+    [
+        ("openai/gpt-4o-mini", ["openai/gpt-4o-mini"]),
+        (" openai/gpt-4o-mini , openai/gpt-4o-mini,anthropic/claude-haiku-4-5, ",
+         ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"]),
+        ("openrouter/meta-llama/llama-4-maverick", ["openrouter/meta-llama/llama-4-maverick"]),
+    ],
+)
+def test_the_models_are_one_list_stripped_with_repeats_dropped(text, expected):
+    assert eval_module.parse_models(text) == expected
+
+
+@pytest.mark.parametrize(
+    ("text", "said"),
+    [
+        ("gpt-4o-mini", "names no provider"),
+        ("openai/", "names no provider"),
+        ("/gpt-4o", "names no provider"),
+        (" , ", "names no model"),
+        (",".join(f"openai/m{n}" for n in range(6)), "at most 5"),
+        ("openai/" + "m" * 200, "longer than 200"),
+    ],
+)
+def test_a_list_amp_would_refuse_is_refused_here_with_an_example(text, said):
+    with pytest.raises(eval_module.EvaluationStoppedError, match=said) as refused:
+        eval_module.parse_models(text)
+    if "provider" in said or "names no model" in said:
+        assert "openai/gpt-4o-mini" in str(refused.value)
+
+
+def test_a_model_without_its_provider_exits_one_before_anything_is_sent(deployed, monkeypatch, capsys):
+    amp = install(monkeypatch, FakeModelsAMP())
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models("gpt-4o-mini")
+
+    assert stopped.value.code == 1 and amp.calls == []
+    assert "provider/model" in capsys.readouterr().out
+
+
+def test_a_failed_comparison_exits_one_with_the_reason(deployed, monkeypatch, capsys):
+    failed = httpx.Response(200, json={"id": "ev-9", "status": "failed", "error": "the deployment answered 500"})
+    install(monkeypatch, FakeModelsAMP(statuses=[failed]))
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models(MODELS)
+
+    assert stopped.value.code == 1
+    assert "Comparison failed: the deployment answered 500" in capsys.readouterr().out
+
+
+def test_done_without_a_comparison_is_a_protocol_error(deployed, monkeypatch, capsys):
+    install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(200, json={"id": "ev-9", "status": "done"})]))
+
+    with pytest.raises(SystemExit) as stopped:
+        eval_module.eval_models(MODELS)
+
+    assert stopped.value.code == 1
+    assert "done without a comparison (protocol error)" in capsys.readouterr().out
+
+
+def test_a_deployment_this_account_may_not_run_is_amps_sentence_alone(deployed, monkeypatch, capsys):
+    """Logging in again changes nothing about a deployment one may not run."""
+    refused = httpx.Response(403, json={"error": "not_allowed", "message": "You may not run poem_creator."})
+    install(monkeypatch, FakeModelsAMP(create=refused))
+
+    with pytest.raises(SystemExit):
+        eval_module.eval_models(MODELS)
+
+    out = capsys.readouterr().out
+    assert "You may not run poem_creator." in out and "crewai login" not in out
+
+
+def test_everything_from_the_wire_prints_literally(deployed, monkeypatch, capsys):
+    hostile = comparison(
+        models=[{"key": "k", "label": "[red]Writer[/red]: [link=https://evil.test]x[/link]", "baseline": True,
+                 "grades": {"goal": 9, "tasks": True, "agents": "5", "tools": None}, "cost_usd": True,
+                 "seconds": -1}],
+        suggestions=[{"subject": "[bold]agent[/bold]", "field": "goal", "problem": "[red]p[/red]",
+                      "change": "[link=https://evil.test]c[/link]"}],
+    )
+    install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(
+        200, json={"id": "ev-9", "status": "done", "url": URL, "comparison": hostile})]))
+
+    eval_module.eval_models(MODELS)
+
+    out = capsys.readouterr().out
+    assert "[red]Writer[/red]: [link=https://evil.test]x[/link]" in out
+    assert "[red]p[/red]" in out and "[link=https://evil.test]c[/link]" in out
+    # Not a grade, a cost or a time: each prints as "—", never as a value.
+    row = next(line for line in out.splitlines() if "Writer" in line)
+    assert "9/5" not in row and "True" not in row and "-1" not in row and "5/5" not in row
+    assert row.count("—") == 6
+
+
+@pytest.mark.parametrize(
+    ("progress", "line"),
+    [
+        ("Setting up the deployment", "Setting up the deployment"),
+        ({"message": "model 2 of 3: mini"}, "model 2 of 3: mini"),
+        ({"event": "configuration_done", "payload": {"index": 0, "total": 2, "key": "base"}},
+         "graded · model 1 of 2 · base"),
+        ({"subject": "the final output"}, "judging the final output"),
+        ({"unknown": "shape"}, None),
+        (["not", "a", "dict"], None),
+    ],
+)
+def test_the_progress_line_says_what_is_there_and_guesses_nothing(progress, line):
+    assert eval_module._progress_line({"progress": progress}) == line
+
+
+def test_the_last_event_stands_in_for_progress():
+    events = [{"event": "configuration_started", "payload": {"index": 0, "total": 2, "key": "base"}},
+              {"event": "judging", "payload": {"subject": "task 'write'"}}]
+
+    assert eval_module._progress_line({"events": events}) == "judging task 'write'"
+
+
+def test_the_cli_maps_models_and_deployment_to_the_comparison(monkeypatch):
+    calls = []
+    monkeypatch.setattr("crewai_cli.cli.eval_models", lambda *args, **kwargs: calls.append((args, kwargs)))
+    monkeypatch.setattr("crewai_cli.cli.eval_crew", lambda **kwargs: calls.append(("run", kwargs)))
+    runner = CliRunner()
+
+    assert runner.invoke(eval_command, ["--models", MODELS]).exit_code == 0
+    assert runner.invoke(eval_command, ["--models", "openai/gpt-4o", "--deployment", DEPLOYMENT]).exit_code == 0
+    assert calls == [((MODELS,), {"deployment_id": None}), (("openai/gpt-4o",), {"deployment_id": DEPLOYMENT})]
+
+    both = runner.invoke(eval_command, ["--models", MODELS, "--run", EXECUTION_ID])
+    alone = runner.invoke(eval_command, ["--deployment", DEPLOYMENT])
+    assert both.exit_code == 2 and "Give one of them" in both.output
+    assert alone.exit_code == 2 and "add --models LIST" in alone.output
+    assert len(calls) == 2
+    assert "--models" in runner.invoke(eval_command, ["--help"]).output
+
+
+def test_a_comparison_leaves_its_criteria_behind_when_the_project_has_none(deployed, monkeypatch, capsys):
+    directory, _ = deployed
+    answer = compared()
+    body = {**answer.json(), "eval_config": '// the criteria\n{"dataset": []}\n'}
+    install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(200, json=body)]))
+
+    eval_module.eval_models(MODELS)
+
+    assert (directory / "eval.jsonc").read_text() == '// the criteria\n{"dataset": []}\n'
+    assert "Wrote eval.jsonc" in capsys.readouterr().out
+
+
+def test_a_comparison_never_overwrites_the_projects_criteria(deployed, monkeypatch, capsys):
+    directory, _ = deployed
+    (directory / "eval.jsonc").write_text("// ours\n{}\n")
+    body = {**compared().json(), "eval_config": "// theirs\n{}\n"}
+    install(monkeypatch, FakeModelsAMP(statuses=[httpx.Response(200, json=body)]))
+
+    eval_module.eval_models(MODELS)
+
+    assert (directory / "eval.jsonc").read_text() == "// ours\n{}\n"
+    assert "Wrote eval.jsonc" not in capsys.readouterr().out
+
+
+@pytest.mark.parametrize(
+    ("typed", "sent"),
+    [
+        ("openai/gpt-4o-mini", "openai/gpt-4o-mini"),
+        ("anthropic/claude-haiku-4-5", "anthropic/claude-haiku-4-5"),
+        ("openrouter/openai/gpt-4o-mini", "openrouter/openai/gpt-4o-mini"),
+        # A customer's own: a fine-tune, a deployment name, a model named after
+        # a public one, an unknown name — by provider only.
+        ("openai/ft:gpt-4o-mini:acme-corp::abc", "openai/other"),
+        ("azure/my-deployment", "azure/other"),
+        ("openai/gpt-4o-acme", "openai/other"),
+        ("ollama/acme-llama", "ollama/other"),
+        # A provider crewAI does not know can name a host.
+        ("llm.acme.internal/llama-3", "other/other"),
+    ],
+)
+def test_usage_stats_name_a_model_only_when_it_is_a_public_one(typed, sent):
+    assert eval_module.telemetry_model_name(typed) == sent
+
+
+def test_without_crewais_catalog_every_model_is_other(monkeypatch):
+    monkeypatch.setattr(eval_module, "_known_models_and_providers", lambda: (frozenset(), frozenset()))
+
+    assert eval_module.telemetry_model_name("openai/gpt-4o-mini") == "other/other"

lib/cli/tests/test_plus_api.py

@@ -33,6 +33,60 @@ class TestPlusAPI(unittest.TestCase):
         )
         self.assertEqual(response, mock_response)
 
+    @patch("crewai_core.plus_api.PlusAPI._make_request")
+    def test_create_evaluation_files_it_under_the_project(self, mock_make_request):
+        self.api.create_evaluation("run-1", eval_config="{}", project_id="proj-1")
+
+        mock_make_request.assert_called_once_with(
+            "POST",
+            "/crewai_plus/api/v1/tracing/evaluations",
+            json={"execution_id": "run-1", "eval_config": "{}", "project_id": "proj-1"},
+            timeout=120.0,
+        )
+
+    @patch("crewai_core.plus_api.PlusAPI._make_request")
+    def test_create_models_evaluation(self, mock_make_request):
+        mock_response = MagicMock()
+        mock_make_request.return_value = mock_response
+
+        response = self.api.create_models_evaluation(
+            ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"],
+            project_id="proj-1",
+            eval_config="// ours\n{}",
+            deployment_id="dep-1",
+        )
+
+        mock_make_request.assert_called_once_with(
+            "POST",
+            "/crewai_plus/api/v1/tracing/evaluations",
+            json={
+                "kind": "models",
+                "project_id": "proj-1",
+                "models": ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"],
+                "eval_config": "// ours\n{}",
+                "deployment_id": "dep-1",
+            },
+            timeout=120.0,
+        )
+        self.assertEqual(response, mock_response)
+
+    @patch("crewai_core.plus_api.PlusAPI._make_request")
+    def test_create_models_evaluation_sends_only_what_it_was_given(
+        self, mock_make_request
+    ):
+        self.api.create_models_evaluation(["openai/gpt-4o-mini"], project_id="proj-1")
+
+        mock_make_request.assert_called_once_with(
+            "POST",
+            "/crewai_plus/api/v1/tracing/evaluations",
+            json={
+                "kind": "models",
+                "project_id": "proj-1",
+                "models": ["openai/gpt-4o-mini"],
+            },
+            timeout=120.0,
+        )
+
     @patch("crewai_core.plus_api.PlusAPI._make_request")
     def test_get_evaluation(self, mock_make_request):
         mock_response = MagicMock()
@@ -252,6 +306,42 @@ class TestPlusAPI(unittest.TestCase):
             "POST", "/crewai_plus/api/v1/crews/test_uuid/deploy"
         )
 
+    @patch("crewai_core.plus_api.PlusAPI._make_request")
+    def test_deploy_sends_the_project_id_when_there_is_one(self, mock_make_request):
+        self.api.deploy_by_name("test_project", project_id="proj-1")
+        mock_make_request.assert_called_with(
+            "POST",
+            "/crewai_plus/api/v1/crews/by-name/test_project/deploy",
+            json={"project_id": "proj-1"},
+        )
+        self.api.deploy_by_uuid("test_uuid", project_id="proj-1")
+        mock_make_request.assert_called_with(
+            "POST",
+            "/crewai_plus/api/v1/crews/test_uuid/deploy",
+            json={"project_id": "proj-1"},
+        )
+
+    @patch("crewai_cli.plus_api.PlusAPI._make_multipart_request")
+    def test_zip_deploys_send_the_project_id_when_there_is_one(
+        self, mock_make_multipart_request
+    ):
+        self.api.create_crew_from_zip("/tmp/test.zip", name="c", project_id="proj-1")
+        mock_make_multipart_request.assert_called_with(
+            "POST",
+            "/crewai_plus/api/v1/crews/zip",
+            zip_file_path="/tmp/test.zip",
+            data={"name": "c", "project_id": "proj-1"},
+            timeout=300,
+        )
+        self.api.update_crew_from_zip("u", "/tmp/test.zip", project_id="proj-1")
+        mock_make_multipart_request.assert_called_with(
+            "POST",
+            "/crewai_plus/api/v1/crews/u/zip_update",
+            zip_file_path="/tmp/test.zip",
+            data={"project_id": "proj-1"},
+            timeout=300,
+        )
+
     @patch("crewai_core.plus_api.PlusAPI._make_request")
     def test_crew_status_by_name(self, mock_make_request):
         self.api.crew_status_by_name("test_project")

lib/crewai-core/pyproject.toml

@@ -13,7 +13,7 @@ dependencies = [
     "httpx~=0.28.1",
     "packaging>=23.0",
     "portalocker~=2.7.0",
-    "pyjwt>=2.13.0,<3",
+    "pyjwt>=2.15.0,<3",
     "pydantic>=2.11.9,<2.13",
     "rich>=13.7.1",
     "opentelemetry-api>=1.42,<2",

lib/crewai-core/src/crewai_core/plus_api.py

@@ -62,6 +62,9 @@ class CrewDeploymentSpec(TypedDict):
 
 class CreateCrewPayload(TypedDict):
     deploy: CrewDeploymentSpec
+    # ``[tool.crewai].project_id``, at the top level as every other deploy
+    # request sends it (AMP reads it there on every route); only when set.
+    project_id: NotRequired[str]
 
 
 class _WithUserIdentifier(TypedDict):
@@ -320,13 +323,29 @@ class PlusAPI:
             params=params or None,
         )
 
-    def deploy_by_name(self, project_name: str) -> httpx.Response:
-        return self._make_request(
-            "POST", f"{self.CREWS_RESOURCE}/by-name/{project_name}/deploy"
-        )
+    def deploy_by_name(
+        self, project_name: str, *, project_id: str | None = None
+    ) -> httpx.Response:
+        """Redeploy by name, with PROJECT_ID (`[tool.crewai].project_id`) when set.
 
-    def deploy_by_uuid(self, uuid: str) -> httpx.Response:
-        return self._make_request("POST", f"{self.CREWS_RESOURCE}/{uuid}/deploy")
+        The body is sent only when there is a project id. An AMP that predates
+        project ids ignores it: this route has read its body through Rails'
+        ``params.permit`` since the v1 API (2024-08), which drops a key it does
+        not name, and AMP never made unpermitted keys an error.
+        """
+        endpoint = f"{self.CREWS_RESOURCE}/by-name/{project_name}/deploy"
+        if project_id:
+            return self._make_request("POST", endpoint, json={"project_id": project_id})
+        return self._make_request("POST", endpoint)
+
+    def deploy_by_uuid(
+        self, uuid: str, *, project_id: str | None = None
+    ) -> httpx.Response:
+        """Redeploy by uuid, with PROJECT_ID as for `deploy_by_name`."""
+        endpoint = f"{self.CREWS_RESOURCE}/{uuid}/deploy"
+        if project_id:
+            return self._make_request("POST", endpoint, json={"project_id": project_id})
+        return self._make_request("POST", endpoint)
 
     def crew_status_by_name(self, project_name: str) -> httpx.Response:
         return self._make_request(
@@ -368,11 +387,18 @@ class PlusAPI:
         *,
         name: str | None = None,
         env: dict[str, str] | None = None,
+        project_id: str | None = None,
     ) -> httpx.Response:
-        """Create a crew deployment from a local project ZIP archive."""
+        """Create a crew deployment from a local project ZIP archive.
+
+        PROJECT_ID is `[tool.crewai].project_id`, so AMP knows which project the
+        deployment runs; sent only when the project has one.
+        """
         data: dict[str, str] = {}
         if name:
             data["name"] = name
+        if project_id:
+            data["project_id"] = project_id
         if env:
             data.update({f"env[{key}]": value for key, value in env.items()})
         return self._make_multipart_request(
@@ -389,9 +415,15 @@ class PlusAPI:
         zip_file_path: str | Path,
         *,
         env: dict[str, str] | None = None,
+        project_id: str | None = None,
     ) -> httpx.Response:
-        """Update an existing crew deployment from a local project ZIP archive."""
+        """Update an existing crew deployment from a local project ZIP archive.
+
+        PROJECT_ID as for `create_crew_from_zip`.
+        """
         data: dict[str, str] = {}
+        if project_id:
+            data["project_id"] = project_id
         if env:
             data.update({f"env[{key}]": value for key, value in env.items()})
         return self._make_multipart_request(

lib/crewai-core/src/crewai_core/telemetry.py

@@ -580,6 +580,11 @@ class Telemetry:
     # its use under somebody else's name.
     FEATURE_ATTRIBUTES: ClassVar[dict[str, frozenset[str]]] = {
         "cli_usage:eval": frozenset({"authenticated"}),
+        # The models compared: a model in crewAI's own catalog by name (anyone
+        # can read it off a price list), any other — a fine-tune, a deployment
+        # name, a self-hosted model — as `<provider>/other`; the CLI decides
+        # (`telemetry_model_name`). Never a run, an output or an organization.
+        "cli_usage:eval_models": frozenset({"authenticated", "models", "models_count"}),
     }
 
     def feature_usage_span(

lib/crewai-core/tests/test_smoke.py

@@ -311,6 +311,45 @@ def test_core_feature_span_keeps_only_what_the_feature_may_send(
     assert "authenticated" not in sent and sent["feature"] == "cli_usage:deploy"
 
 
+def test_core_models_span_carries_the_models_compared_and_nothing_else(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """`cli_usage:eval_models` names the models and how many — provider/model
+    names, not a run — and still drops an id or run content."""
+    from crewai_core.telemetry import Telemetry
+
+    Telemetry._instance = None
+    monkeypatch.delenv("OTEL_SDK_DISABLED", raising=False)
+    monkeypatch.delenv("CREWAI_DISABLE_TELEMETRY", raising=False)
+    monkeypatch.delenv("CREWAI_DISABLE_TRACKING", raising=False)
+
+    tracer = Mock()
+    span = Mock()
+    tracer.start_span.return_value = span
+    monkeypatch.setattr(
+        "crewai_core.telemetry.TracerProvider",
+        lambda **_kwargs: Mock(get_tracer=Mock(return_value=tracer)),
+    )
+
+    Telemetry().feature_usage_span(
+        "cli_usage:eval_models",
+        {
+            "authenticated": "true",
+            "models": "openai/gpt-4o-mini,anthropic/claude-haiku-4-5",
+            "models_count": "2",
+            "evaluation_id": "ev-1",
+            "organization_id": "org-1",
+        },
+    )
+
+    sent = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}
+    assert sent["feature"] == "cli_usage:eval_models"
+    assert sent["authenticated"] == "true"
+    assert sent["models"] == "openai/gpt-4o-mini,anthropic/claude-haiku-4-5"
+    assert sent["models_count"] == "2"
+    assert "evaluation_id" not in sent and "organization_id" not in sent
+
+
 def test_core_telemetry_records_flow_creation_version(
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:

lib/crewai/pyproject.toml

@@ -24,12 +24,12 @@ dependencies = [
     "opentelemetry-sdk>=1.42,<2",
     "opentelemetry-exporter-otlp-proto-http>=1.42,<2",
     # Data Handling
-    "chromadb~=1.1.0",
+    "chromadb~=1.5.9",
     "tokenizers>=0.21,<1",
     "openpyxl~=3.1.5",
     # Authentication and Security
     "python-dotenv>=1.2.2,<2",
-    "pyjwt>=2.13.0,<3",
+    "pyjwt>=2.15.0,<3",
     # Configuration and Utils
     "click>=8.1.7,<9",
     "appdirs~=1.4.4",
@@ -90,7 +90,7 @@ voyageai = [
     "voyageai~=0.3.5",
 ]
 litellm = [
-    "litellm>=1.84.0,<2",
+    "litellm>=1.88.6,<2",
 ]
 bedrock = [
     "boto3~=1.43.46",

lib/crewai/src/crewai/agent/core.py

@@ -79,7 +79,7 @@ from crewai.hooks.dispatch import HookAborted
 from crewai.knowledge.knowledge import Knowledge
 from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSource
 from crewai.lite_agent_output import LiteAgentOutput
-from crewai.llm_overlay import active as overlay_active, overlay_model_for
+from crewai.llm_overlay import active as overlay_active
 from crewai.llms.base_llm import BaseLLM
 from crewai.mcp.config import MCPServerConfig
 from crewai.rag.embeddings.types import EmbedderConfig
@@ -114,7 +114,7 @@ from crewai.utilities.env import get_env_context
 from crewai.utilities.guardrail import process_guardrail, serialize_guardrail_for_json
 from crewai.utilities.guardrail_types import GuardrailCallable, GuardrailType
 from crewai.utilities.i18n import I18N_DEFAULT
-from crewai.utilities.llm_utils import create_llm, create_llm_like
+from crewai.utilities.llm_utils import create_llm, overlay_llm_for
 from crewai.utilities.prompts import Prompts, StandardPromptResult, SystemPromptResult
 from crewai.utilities.pydantic_schema_utils import generate_model_description
 from crewai.utilities.string_utils import sanitize_tool_name
@@ -420,13 +420,7 @@ class Agent(BaseAgent):
         else:
             declared = create_llm(self.llm)
             self._declared_llm = declared
-            overlay_model = overlay_model_for(self.role)
-            if overlay_model:
-                self.llm = self._overlay_built = create_llm_like(
-                    overlay_model, declared
-                )
-            else:
-                self.llm = declared
+            self.llm = self._resolved_overlay(declared)
             self._overlay_read = True
         if self.function_calling_llm and not isinstance(
             self.function_calling_llm, BaseLLM
@@ -523,15 +517,22 @@ class Agent(BaseAgent):
         if self.role == role_before or overlay_active.get() is None:
             return
         previous = self.llm
-        declared = self._declared_now()
-        overlay_model = overlay_model_for(self.role)
-        if overlay_model:
-            self.llm = self._overlay_built = create_llm_like(overlay_model, declared)
-        else:
-            self.llm = declared
+        self.llm = self._resolved_overlay(self._declared_now())
         if isinstance(previous, BaseLLM) and isinstance(self.llm, BaseLLM):
             self.llm.stream = previous.stream
 
+    def _resolved_overlay(self, declared: BaseLLM | None) -> BaseLLM | None:
+        """The llm ``llm_overlay`` puts this agent on, given its ``declared`` one.
+
+        The role's key wins over the model keys; with neither, ``declared``.
+        What the overlay built is remembered, so a later miss can tell it from
+        an llm the caller assigned.
+        """
+        resolved = overlay_llm_for(self.role, declared)
+        if resolved is not declared:
+            self._overlay_built = resolved
+        return resolved
+
     def _declared_now(self) -> BaseLLM | None:
         """The llm this agent is declared with, as of now.
 

lib/crewai/src/crewai/execution.py

@@ -18,6 +18,7 @@ from __future__ import annotations
 from contextlib import ExitStack
 import contextvars
 from dataclasses import dataclass
+import logging
 import os
 import sys
 from types import TracebackType
@@ -29,6 +30,9 @@ if TYPE_CHECKING:
     from crewai.telemetry.tracing.session import TraceSession
 
 
+logger = logging.getLogger(__name__)
+
+
 @dataclass
 class ExecutionTrace:
     """Trace lifetime that can be rebound between deferred conversational turns."""
@@ -143,22 +147,38 @@ def _start_tracing(execution_uuid: str, tracing: bool | None) -> None:
     from crewai.telemetry.tracing.grants import (
         GrantSpanExporter,
         TraceGrantClient,
-        tracing_credential,
+        resolve_tracing_credential,
     )
     from crewai.telemetry.tracing.session import TraceSession
 
     stack = ExitStack()
     # First-run discovery is local even if CLI credentials happen to exist.
-    amp_credential = tracing_credential() if enabled else None
-    if amp_credential is None:
+    # The credential and its source are resolved ONCE: the pair that is sent is
+    # the pair a refusal names, whatever the environment says afterwards.
+    resolved = resolve_tracing_credential() if enabled else None
+    if resolved is None:
         from crewai.telemetry.tracing.ephemeral import ephemeral_tracing
 
         session = stack.enter_context(
             ephemeral_tracing(execution_uuid, first_time=not enabled)
         )
     else:
-        client = TraceGrantClient(amp_credential)
-        grant = client.create(execution_uuid)
+        from crewai.telemetry.tracing.grants import TraceGrantError
+
+        credential_source, amp_credential = resolved
+        try:
+            # The constructor refuses a blank credential with the same error, so
+            # it sits inside the same boundary as the grant request.
+            client = TraceGrantClient(amp_credential)
+            grant = client.create(execution_uuid)
+        except TraceGrantError as error:
+            # A trace is a record of the run, not a condition of it: a login
+            # that expired or a token that was revoked must not take the run
+            # down with it. The run goes on untraced and says so — never falls
+            # back to an anonymous upload of a run whose owner is logged in.
+            logger.warning(_untraced_because(error, credential_source))
+            stack.close()
+            return
         exporter = GrantSpanExporter(client, grant)
         session = TraceSession(grant.execution_uuid, [exporter])
 
@@ -170,6 +190,38 @@ def _start_tracing(execution_uuid: str, tracing: bool | None) -> None:
     _activate_tracing(ExecutionTrace(session, stack))
 
 
+# What each credential is called, and what fixes it when AMP refuses it: the one
+# that was sent, never a different one — refreshing a login does nothing for a
+# rejected CREWAI_USER_PAT.
+_CREDENTIAL_FIX = {
+    "pat": (
+        "the CREWAI_USER_PAT token",
+        "Replace it with a valid personal access token",
+    ),
+    "integration": (
+        "the platform integration token",
+        "Check the integration token this environment is given",
+    ),
+    "login": ("the saved login", "Run `crewai login` again"),
+}
+
+
+def _untraced_because(error: Exception, source: str | None) -> str:
+    """The warning for a run AMP would not grant a trace to, naming the
+    credential it refused and the fix for that one."""
+    status = getattr(error, "status_code", None)
+    if status in (401, 403):
+        name, fix = _CREDENTIAL_FIX.get(source or "", ("the credential", "Check it"))
+        return (
+            f"This run is not traced: CrewAI AMP refused {name} (HTTP {status}). "
+            f"{fix} to trace your runs."
+        )
+    return (
+        f"This run is not traced: CrewAI AMP could not grant a trace "
+        f"({f'HTTP {status}' if status else error}). The run itself is unaffected."
+    )
+
+
 def _activate_tracing(tracing: ExecutionTrace) -> None:
     activation = ExitStack()
     activation.enter_context(tracing.session.activate())

lib/crewai/src/crewai/lite_agent.py

@@ -69,7 +69,6 @@ from crewai.hooks.types import (
 )
 from crewai.lite_agent_output import LiteAgentOutput
 from crewai.llm import LLM
-from crewai.llm_overlay import overlay_model_for
 from crewai.llms.base_llm import BaseLLM
 from crewai.tools.base_tool import BaseTool
 from crewai.tools.structured_tool import CrewStructuredTool
@@ -102,7 +101,7 @@ from crewai.utilities.converter import (
 from crewai.utilities.guardrail import process_guardrail, serialize_guardrail_for_json
 from crewai.utilities.guardrail_types import GuardrailCallable, GuardrailType
 from crewai.utilities.i18n import I18N_DEFAULT
-from crewai.utilities.llm_utils import create_llm, create_llm_like
+from crewai.utilities.llm_utils import create_llm, overlay_llm_for
 from crewai.utilities.pydantic_schema_utils import (
     generate_model_description,
     serialize_model_class,
@@ -320,11 +319,7 @@ class LiteAgent(FlowTrackable, BaseModel):
     @model_validator(mode="after")
     def setup_llm(self) -> Self:
         """Set up the LLM and other components after initialization."""
-        declared = create_llm(self.llm)
-        overlay_model = overlay_model_for(self.role)
-        self.llm = (
-            create_llm_like(overlay_model, declared) if overlay_model else declared
-        )
+        self.llm = overlay_llm_for(self.role, create_llm(self.llm))
         if not isinstance(self.llm, BaseLLM):
             raise ValueError(
                 f"Expected LLM instance of type BaseLLM, got {type(self.llm).__name__}"

lib/crewai/src/crewai/llm.py

@@ -1,7 +1,8 @@
 from __future__ import annotations
 
 from collections import defaultdict
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
+from contextvars import ContextVar
 from datetime import datetime
 import json
 import logging
@@ -33,6 +34,7 @@ from crewai.events.types.tool_usage_events import (
     ToolUsageStartedEvent,
 )
 from crewai.hooks.dispatch import HookAborted
+from crewai.llm_overlay import overlay_model_for_model
 from crewai.llms._finish_reason_utils import extract_choices_finish_reason_and_id
 from crewai.llms.base_llm import (
     BaseLLM,
@@ -48,6 +50,13 @@ from crewai.llms.constants import (
     GEMINI_MODELS,
     OPENAI_MODELS,
 )
+from crewai.llms.context_window import (
+    AZURE_OPENAI_CONTEXT_WINDOWS,
+    DEFAULT_CONTEXT_WINDOW_SIZE,
+    LLM_CONTEXT_WINDOW_SIZES,
+    OPENAI_CONTEXT_WINDOWS,
+    resolve_context_window_size,
+)
 from crewai.utilities import InternalInstructor
 from crewai.utilities.exceptions.context_window_exceeding_exception import (
     LLMContextLengthExceededError,
@@ -81,6 +90,10 @@ logger = logging.getLogger(__name__)
 _litellm_loaded = False
 LITELLM_AVAILABLE = False
 
+# The instance ``LLM.__new__`` just built on a model ``llm_overlay`` mapped to,
+# whose ``__init__`` call with the caller's arguments must be skipped.
+_overlay_built: ContextVar[LLM | None] = ContextVar("llm_overlay_built", default=None)
+
 if TYPE_CHECKING:
     import litellm
     from litellm.litellm_core_utils.get_supported_openai_params import (
@@ -164,171 +177,8 @@ def _ensure_litellm() -> bool:
     return LITELLM_AVAILABLE
 
 
-MIN_CONTEXT: Final[int] = 1024
-MAX_CONTEXT: Final[int] = 2097152  # Current max from gemini-1.5-pro
 ANTHROPIC_PREFIXES: Final[tuple[str, str, str]] = ("anthropic/", "claude-", "claude/")
 
-LLM_CONTEXT_WINDOW_SIZES: Final[dict[str, int]] = {
-    "gpt-4": 8192,
-    "gpt-4o": 128000,
-    "gpt-4o-mini": 128000,
-    "gpt-5.4-mini": 200000,
-    "gpt-5.6": 1050000,  # sol, terra, luna, and the gpt-5.6 alias
-    "gpt-4-turbo": 128000,
-    "gpt-4.1": 1047576,  # Based on official docs
-    "gpt-4.1-mini-2025-04-14": 1047576,
-    "gpt-4.1-nano-2025-04-14": 1047576,
-    "o1-preview": 128000,
-    "o1-mini": 128000,
-    "o3-mini": 200000,
-    "o4-mini": 200000,
-    "gemini-3.8-flash": 1048576,
-    "gemini-3-pro-preview": 1048576,
-    "gemini-2.0-flash": 1048576,
-    "gemini-2.0-flash-thinking-exp-01-21": 32768,
-    "gemini-2.0-flash-lite-001": 1048576,
-    "gemini-2.0-flash-001": 1048576,
-    "gemini-2.5-flash-preview-04-17": 1048576,
-    "gemini-2.5-pro-exp-03-25": 1048576,
-    "gemini-1.5-pro": 2097152,
-    "gemini-1.5-flash": 1048576,
-    "gemini-1.5-flash-8b": 1048576,
-    "gemini/gemma-3-1b-it": 32000,
-    "gemini/gemma-3-4b-it": 128000,
-    "gemini/gemma-3-12b-it": 128000,
-    "gemini/gemma-3-27b-it": 128000,
-    "deepseek-chat": 128000,
-    "gemma2-9b-it": 8192,
-    "gemma-7b-it": 8192,
-    "llama3-groq-70b-8192-tool-use-preview": 8192,
-    "llama3-groq-8b-8192-tool-use-preview": 8192,
-    "llama-3.1-70b-versatile": 131072,
-    "llama-3.1-8b-instant": 131072,
-    "llama-3.2-1b-preview": 8192,
-    "llama-3.2-3b-preview": 8192,
-    "llama-3.2-11b-text-preview": 8192,
-    "llama-3.2-90b-text-preview": 8192,
-    "llama3-70b-8192": 8192,
-    "llama3-8b-8192": 8192,
-    "mixtral-8x7b-32768": 32768,
-    "llama-3.3-70b-versatile": 128000,
-    "llama-3.3-70b-instruct": 128000,
-    "Meta-Llama-3.3-70B-Instruct": 131072,
-    "QwQ-32B-Preview": 8192,
-    "Qwen2.5-72B-Instruct": 8192,
-    "Qwen2.5-Coder-32B-Instruct": 8192,
-    "Meta-Llama-3.1-405B-Instruct": 8192,
-    "Meta-Llama-3.1-70B-Instruct": 131072,
-    "Meta-Llama-3.1-8B-Instruct": 131072,
-    "Llama-3.2-90B-Vision-Instruct": 16384,
-    "Llama-3.2-11B-Vision-Instruct": 16384,
-    "Meta-Llama-3.2-3B-Instruct": 4096,
-    "Meta-Llama-3.2-1B-Instruct": 16384,
-    "us.amazon.nova-pro-v1:0": 300000,
-    "us.amazon.nova-micro-v1:0": 128000,
-    "us.amazon.nova-lite-v1:0": 300000,
-    "us.anthropic.claude-opus-4-7": 1000000,
-    "us.anthropic.claude-sonnet-4-6": 1000000,
-    "us.anthropic.claude-opus-4-6-v1": 1000000,
-    "us.anthropic.claude-opus-4-5-20251101-v1:0": 200000,
-    "us.anthropic.claude-haiku-4-5-20251001-v1:0": 200000,
-    "us.anthropic.claude-sonnet-4-5-20250929-v1:0": 200000,
-    "us.anthropic.claude-opus-4-1-20250805-v1:0": 200000,
-    "us.anthropic.claude-opus-4-20250514-v1:0": 200000,
-    "us.anthropic.claude-sonnet-4-20250514-v1:0": 200000,
-    "us.anthropic.claude-3-5-sonnet-20240620-v1:0": 200000,
-    "us.anthropic.claude-3-5-haiku-20241022-v1:0": 200000,
-    "us.anthropic.claude-3-5-sonnet-20241022-v2:0": 200000,
-    "us.anthropic.claude-3-7-sonnet-20250219-v1:0": 200000,
-    "us.anthropic.claude-3-sonnet-20240229-v1:0": 200000,
-    "us.anthropic.claude-3-opus-20240229-v1:0": 200000,
-    "us.anthropic.claude-3-haiku-20240307-v1:0": 200000,
-    "us.meta.llama3-2-11b-instruct-v1:0": 128000,
-    "us.meta.llama3-2-3b-instruct-v1:0": 131000,
-    "us.meta.llama3-2-90b-instruct-v1:0": 128000,
-    "us.meta.llama3-2-1b-instruct-v1:0": 131000,
-    "us.meta.llama3-1-8b-instruct-v1:0": 128000,
-    "us.meta.llama3-1-70b-instruct-v1:0": 128000,
-    "us.meta.llama3-3-70b-instruct-v1:0": 128000,
-    "us.meta.llama3-1-405b-instruct-v1:0": 128000,
-    "eu.anthropic.claude-3-5-sonnet-20240620-v1:0": 200000,
-    "eu.anthropic.claude-3-sonnet-20240229-v1:0": 200000,
-    "eu.anthropic.claude-3-haiku-20240307-v1:0": 200000,
-    "eu.anthropic.claude-opus-4-7": 1000000,
-    "eu.anthropic.claude-sonnet-4-6": 1000000,
-    "eu.anthropic.claude-opus-4-6-v1": 1000000,
-    "eu.anthropic.claude-opus-4-5-20251101-v1:0": 200000,
-    "eu.anthropic.claude-haiku-4-5-20251001-v1:0": 200000,
-    "eu.anthropic.claude-sonnet-4-5-20250929-v1:0": 200000,
-    "eu.anthropic.claude-opus-4-1-20250805-v1:0": 200000,
-    "eu.anthropic.claude-opus-4-20250514-v1:0": 200000,
-    "eu.anthropic.claude-sonnet-4-20250514-v1:0": 200000,
-    "eu.meta.llama3-2-3b-instruct-v1:0": 131000,
-    "eu.meta.llama3-2-1b-instruct-v1:0": 131000,
-    "apac.anthropic.claude-3-5-sonnet-20240620-v1:0": 200000,
-    "apac.anthropic.claude-3-5-sonnet-20241022-v2:0": 200000,
-    "apac.anthropic.claude-3-sonnet-20240229-v1:0": 200000,
-    "apac.anthropic.claude-3-haiku-20240307-v1:0": 200000,
-    "apac.anthropic.claude-opus-4-7": 1000000,
-    "apac.anthropic.claude-sonnet-4-6": 1000000,
-    "apac.anthropic.claude-opus-4-6-v1": 1000000,
-    "apac.anthropic.claude-opus-4-5-20251101-v1:0": 200000,
-    "apac.anthropic.claude-haiku-4-5-20251001-v1:0": 200000,
-    "apac.anthropic.claude-sonnet-4-5-20250929-v1:0": 200000,
-    "apac.anthropic.claude-opus-4-1-20250805-v1:0": 200000,
-    "apac.anthropic.claude-opus-4-20250514-v1:0": 200000,
-    "apac.anthropic.claude-sonnet-4-20250514-v1:0": 200000,
-    "amazon.nova-pro-v1:0": 300000,
-    "amazon.nova-micro-v1:0": 128000,
-    "amazon.nova-lite-v1:0": 300000,
-    "anthropic.claude-opus-4-7": 1000000,
-    "anthropic.claude-sonnet-4-6": 1000000,
-    "anthropic.claude-opus-4-6-v1": 1000000,
-    "anthropic.claude-opus-4-5-20251101-v1:0": 200000,
-    "anthropic.claude-haiku-4-5-20251001-v1:0": 200000,
-    "anthropic.claude-sonnet-4-5-20250929-v1:0": 200000,
-    "anthropic.claude-opus-4-1-20250805-v1:0": 200000,
-    "anthropic.claude-opus-4-20250514-v1:0": 200000,
-    "anthropic.claude-sonnet-4-20250514-v1:0": 200000,
-    "anthropic.claude-3-5-sonnet-20240620-v1:0": 200000,
-    "anthropic.claude-3-5-haiku-20241022-v1:0": 200000,
-    "anthropic.claude-3-5-sonnet-20241022-v2:0": 200000,
-    "anthropic.claude-3-7-sonnet-20250219-v1:0": 200000,
-    "anthropic.claude-3-sonnet-20240229-v1:0": 200000,
-    "anthropic.claude-3-opus-20240229-v1:0": 200000,
-    "anthropic.claude-3-haiku-20240307-v1:0": 200000,
-    "anthropic.claude-v2:1": 200000,
-    "anthropic.claude-v2": 100000,
-    "anthropic.claude-instant-v1": 100000,
-    "meta.llama3-1-405b-instruct-v1:0": 128000,
-    "meta.llama3-1-70b-instruct-v1:0": 128000,
-    "meta.llama3-1-8b-instruct-v1:0": 128000,
-    "meta.llama3-70b-instruct-v1:0": 8000,
-    "meta.llama3-8b-instruct-v1:0": 8000,
-    "amazon.titan-text-lite-v1": 4000,
-    "amazon.titan-text-express-v1": 8000,
-    "cohere.command-text-v14": 4000,
-    "ai21.j2-mid-v1": 8191,
-    "ai21.j2-ultra-v1": 8191,
-    "ai21.jamba-instruct-v1:0": 256000,
-    "mistral.mistral-7b-instruct-v0:2": 32000,
-    "mistral.mixtral-8x7b-instruct-v0:1": 32000,
-    "mistral-tiny": 32768,
-    "mistral-small-latest": 32768,
-    "mistral-medium-latest": 32768,
-    "mistral-large-latest": 32768,
-    "mistral-large-2407": 32768,
-    "mistral-large-2402": 32768,
-    "mistral/mistral-tiny": 32768,
-    "mistral/mistral-small-latest": 32768,
-    "mistral/mistral-medium-latest": 32768,
-    "mistral/mistral-large-latest": 32768,
-    "mistral/mistral-large-2407": 32768,
-    "mistral/mistral-large-2402": 32768,
-}
-
-DEFAULT_CONTEXT_WINDOW_SIZE: Final[int] = 8192
-CONTEXT_WINDOW_USAGE_RATIO: Final[float] = 0.85
 SUPPORTED_NATIVE_PROVIDERS: Final[list[str]] = [
     "openai",
     "anthropic",
@@ -350,6 +200,30 @@ SUPPORTED_NATIVE_PROVIDERS: Final[list[str]] = [
 ]
 
 
+# The provider a ``<prefix>/<model>`` string routes to, by prefix: the names
+# crewAI accepts for each native provider. ``llm_overlay`` compares model
+# names through the same table, so ``google/x`` and ``gemini/x`` are one model.
+PROVIDER_ALIASES: Final[dict[str, str]] = {
+    "openai": "openai",
+    "anthropic": "anthropic",
+    "claude": "anthropic",
+    "azure": "azure",
+    "azure_openai": "azure",
+    "google": "gemini",
+    "gemini": "gemini",
+    "bedrock": "bedrock",
+    "aws": "bedrock",
+    "openrouter": "openrouter",
+    "deepseek": "deepseek",
+    "ollama": "ollama",
+    "ollama_chat": "ollama_chat",
+    "hosted_vllm": "hosted_vllm",
+    "cerebras": "cerebras",
+    "dashscope": "dashscope",
+    "snowflake": "snowflake",
+}
+
+
 class Delta(TypedDict):
     content: str | None
     role: str | None
@@ -422,6 +296,11 @@ class LLM(BaseLLM):
         if not model or not isinstance(model, str):
             raise ValueError("Model must be a non-empty string")
 
+        if cls is LLM:
+            mapped = cls._overlay_mapped(model, is_litellm, kwargs)
+            if mapped is not None:
+                return mapped
+
         route = cls._resolve_route(model, kwargs)
         if route.native_class is not None and not is_litellm:
             try:
@@ -466,6 +345,39 @@ class LLM(BaseLLM):
 
         return object.__new__(cls)
 
+    def __init__(self, model: str, is_litellm: bool = False, **kwargs: Any) -> None:
+        # Python calls ``__init__`` on whatever ``__new__`` returns when it is an
+        # ``LLM``; an instance ``llm_overlay`` built on a mapped model is already
+        # initialized, and initializing it again with the caller's arguments
+        # would put the declared model back.
+        if _overlay_built.get() is self:
+            _overlay_built.set(None)
+            return
+        super().__init__(model=model, is_litellm=is_litellm, **kwargs)
+
+    @classmethod
+    def _overlay_mapped(
+        cls, model: str, is_litellm: bool, kwargs: dict[str, Any]
+    ) -> LLM | None:
+        """``LLM(model, **kwargs)`` built on the model an ``llm_overlay`` key maps it to.
+
+        ``None`` when no model key of the active overlay matches ``model``. The
+        provider a bare model routes to is put beside it, so a key written
+        ``model:openai/gpt-4o`` matches ``LLM(model="gpt-4o")``.
+        """
+        provider = kwargs.get("provider")
+        if not provider and "/" not in model:
+            provider = cls._infer_provider_from_model(model)
+        mapped_model = overlay_model_for_model(model, provider)
+        if mapped_model is None:
+            return None
+        from crewai.utilities.llm_utils import create_llm_from_kwargs_like
+
+        built = create_llm_from_kwargs_like(mapped_model, model, kwargs, is_litellm)
+        if isinstance(built, LLM):
+            _overlay_built.set(built)
+        return cast(LLM, built)
+
     @classmethod
     def _matches_provider_pattern(cls, model: str, provider: str) -> bool:
         """Check if a model name matches provider-specific patterns.
@@ -572,27 +484,7 @@ class LLM(BaseLLM):
         elif "/" in model:
             prefix, _, model_part = model.partition("/")
 
-            provider_mapping = {
-                "openai": "openai",
-                "anthropic": "anthropic",
-                "claude": "anthropic",
-                "azure": "azure",
-                "azure_openai": "azure",
-                "google": "gemini",
-                "gemini": "gemini",
-                "bedrock": "bedrock",
-                "aws": "bedrock",
-                "openrouter": "openrouter",
-                "deepseek": "deepseek",
-                "ollama": "ollama",
-                "ollama_chat": "ollama_chat",
-                "hosted_vllm": "hosted_vllm",
-                "cerebras": "cerebras",
-                "dashscope": "dashscope",
-                "snowflake": "snowflake",
-            }
-
-            canonical_provider = provider_mapping.get(prefix.lower())
+            canonical_provider = PROVIDER_ALIASES.get(prefix.lower())
 
             valid_native_model = bool(
                 canonical_provider
@@ -2541,6 +2433,22 @@ class LLM(BaseLLM):
             return remainder
         return self.model
 
+    def _context_window_sizes(self) -> Mapping[str, int]:
+        """Pick a provider catalog when the model id is provider-qualified.
+
+        The merged catalog lets Azure's ``gpt-4`` override OpenAI's. Qualified
+        LiteLLM ids must keep the provider that was named.
+        """
+        prefix, separator, _remainder = self.model.partition("/")
+        if not separator:
+            return LLM_CONTEXT_WINDOW_SIZES
+        provider = prefix.lower()
+        if provider == "openai":
+            return OPENAI_CONTEXT_WINDOWS
+        if provider in {"azure", "azure_openai"}:
+            return AZURE_OPENAI_CONTEXT_WINDOWS
+        return LLM_CONTEXT_WINDOW_SIZES
+
     def get_context_window_size(self) -> int:
         """
         Returns the context window size, using 75% of the maximum to avoid
@@ -2552,22 +2460,12 @@ class LLM(BaseLLM):
         if self.context_window_size != 0:
             return self.context_window_size
 
-        min_context = 1024
-        max_context = 2097152  # Current max from gemini-1.5-pro
-
-        for key, value in LLM_CONTEXT_WINDOW_SIZES.items():
-            if value < min_context or value > max_context:
-                raise ValueError(
-                    f"Context window for {key} must be between {min_context} and {max_context}"
-                )
-
-        self.context_window_size = int(
-            DEFAULT_CONTEXT_WINDOW_SIZE * CONTEXT_WINDOW_USAGE_RATIO
+        self.context_window_size = resolve_context_window_size(
+            self._context_window_model_name(),
+            self._context_window_sizes(),
+            default=DEFAULT_CONTEXT_WINDOW_SIZE,
+            extra_names=(self.model,),
         )
-        model_name = self._context_window_model_name()
-        for key, value in LLM_CONTEXT_WINDOW_SIZES.items():
-            if model_name.startswith(key) or self.model.startswith(key):
-                self.context_window_size = int(value * CONTEXT_WINDOW_USAGE_RATIO)
         return self.context_window_size
 
     @staticmethod

lib/crewai/src/crewai/llm_overlay.py

@@ -1,10 +1,37 @@
-"""Process-context ``role -> model`` overlay for agents.
+"""Process-context ``role -> model`` and ``model -> model`` overlay.
 
-``llm_overlay`` lets a per-run caller make specific agents use a different
-model without editing the code that builds them. While the block is active,
-an ``Agent`` or ``LiteAgent`` whose ``role`` is a key of the mapping is built
-with the mapped model instead of its declared ``llm``. Roles that are not
-keys, and every agent built outside the block, keep their own model.
+``llm_overlay`` lets a per-run caller make specific agents, or specific
+models, use a different model without editing the code that builds them.
+While the block is active, an ``Agent`` or ``LiteAgent`` whose ``role`` is a
+key of the mapping is built with the mapped model instead of its declared
+``llm``. Roles that are not keys, and every agent built outside the block,
+keep their own model.
+
+A key that starts with :data:`MODEL_KEY_PREFIX` names a model instead of a
+role: ``"model:openai/gpt-4o"`` maps every LLM built from that model string
+inside the block, and ``"model:*"`` every LLM built from any model string.
+That is what reaches an LLM no role names — a flow step's own
+``LLM(model="openai/gpt-4o").call(...)`` — and the simplest way to say "this
+whole run on model X". A model key is read wherever an LLM is built from a
+model string: ``LLM(model=...)`` (and so ``create_llm`` and an agent's
+``llm="..."``, which go through it), and an agent's declared ``llm`` instance
+when it was built outside the block. A model is compared as the caller wrote
+it and without the provider prefix native providers strip, so
+``"model:openai/gpt-4o"`` matches ``LLM(model="gpt-4o")`` and an instance
+whose ``model`` reads ``"gpt-4o"``, and ``"model:gpt-4o"`` matches
+``LLM(model="openai/gpt-4o")``; an exact key wins over a stripped one, and
+both over ``"model:*"``. Only the model's own provider prefix is ever
+stripped: an aggregator's route such as ``openrouter/openai/gpt-4o`` is that
+aggregator's model and matches only a key naming it whole. The mapped model is built once, never looked up
+again: ``{"model:a": "b", "model:b": "c"}`` puts an ``a`` on ``b``. For an
+agent a role key wins over every model key, so ``{"Researcher": x,
+"model:*": y}`` runs the Researcher on ``x`` and everything else on ``y`` —
+and ``x`` is built like the llm the caller declared, even when a model key
+already mapped that llm as it was built inside the block.
+The settings the caller passed to ``LLM(...)`` are carried to the mapped model
+by the same rule as a declared ``llm``'s (below). Only ``LLM`` itself is
+mapped: a subclass of it, or a provider class built directly, is the caller's
+own choice of class and keeps its model.
 
 Roles are matched exactly, by the text they have when the overlay is read;
 only whitespace around a role or a key is ignored, so a role a YAML file
@@ -46,6 +73,9 @@ registers an agent when it first emits) does not read it again, so an agent
 built outside a block keeps its model through a kickoff inside one.
 
 Example:
+    >>> with llm_overlay({"model:*": "openai/gpt-4o-mini"}):
+    ...     flow.kickoff()  # every LLM built in the run is on gpt-4o-mini
+
     >>> with llm_overlay({"Researcher": "openai/gpt-4o"}):
     ...     crew = build_crew()  # the Researcher agent is built on gpt-4o
     ...     crew.kickoff()
@@ -60,24 +90,57 @@ from __future__ import annotations
 from collections.abc import Iterator
 from contextlib import contextmanager
 from contextvars import ContextVar
+from typing import Any, Final, TypeVar
+import weakref
+
+
+MODEL_KEY_PREFIX: Final[str] = "model:"
+"""A key starting with this names a model (``"model:openai/gpt-4o"``), not a role."""
+
+_T = TypeVar("_T")
+
+ANY_MODEL: Final[str] = "*"
+"""The model part of the key that maps every LLM built from a model string."""
 
 
 active: ContextVar[dict[str, str] | None] = ContextVar(
     "llm_overlay_active", default=None
 )
 
+# Set while the overlay builds the model it mapped to, so that build is not
+# mapped again (a chain of model keys, or ``model:*`` under a role's model).
+_building_mapped: ContextVar[bool] = ContextVar(
+    "llm_overlay_building_mapped", default=False
+)
+
+# The LLMs a model key already mapped, by id. An agent reading its declared
+# ``llm`` skips one of these: its model is the mapped one, never a key again.
+_mapped_llms: weakref.WeakValueDictionary[int, Any] = weakref.WeakValueDictionary()
+
+# What the caller declared for an LLM the overlay built: the declared instance,
+# or ``LLM(model, **kwargs)``'s ``(model, kwargs, is_litellm)``. Dropped with
+# the LLM it describes.
+Declared = Any
+_declared_before: dict[int, Declared] = {}
+
 
 @contextmanager
 def llm_overlay(mapping: dict[str, str] | None) -> Iterator[None]:
-    """Route agent roles to models for the duration of the block.
+    """Route agent roles and models to models for the duration of the block.
 
     Args:
-        mapping: ``{role: model}`` for the block; ``None`` clears any active
-            overlay for the block. Whitespace around each role is dropped from
-            the copy the block uses, so a key matches a role that differs from
-            it only by its surrounding whitespace; the mapping passed in is
-            left as it is. The previous value is always restored on exit,
-            including when the block raises.
+        mapping: ``{role: model}`` and ``{"model:<model>": model}`` (or
+            ``{"model:*": model}``) for the block; ``None`` clears any active
+            overlay for the block. Whitespace around each role, and around the
+            model a model key names, is dropped from the copy the block uses,
+            so a key matches a role that differs from it only by its
+            surrounding whitespace; the mapping passed in is left as it is. The
+            previous value is always restored on exit, including when the
+            block raises.
+
+    Raises:
+        ValueError: Two keys name one role or one model with different
+            models, or a model key names no model.
     """
     token = active.set(_stripped(mapping))
     try:
@@ -90,7 +153,8 @@ def overlay_model_for(role: str | None) -> str | None:
     """The model the active overlay assigns to ``role``.
 
     Whitespace around ``role`` is ignored, as it was around the keys when the
-    overlay was set. An empty or ``None`` role matches nothing.
+    overlay was set. An empty or ``None`` role matches nothing, and a role is
+    never matched against a model key.
 
     Returns:
         The mapped model string, or ``None`` when no overlay is active or
@@ -100,22 +164,166 @@ def overlay_model_for(role: str | None) -> str | None:
     if not mapping or role is None:
         return None
     role = role.strip()
-    return mapping.get(role) if role else None
+    if not role or role.startswith(MODEL_KEY_PREFIX):
+        return None
+    return mapping.get(role)
+
+
+def overlay_model_for_model(
+    model: str | None, provider: str | None = None
+) -> str | None:
+    """The model the active overlay's model keys assign to an LLM on ``model``.
+
+    ``model`` is compared as written and, when a native provider strips its own
+    prefix, without it: ``openai/gpt-4o`` also as ``gpt-4o``, and ``gpt-4o``
+    routed to ``provider`` also as ``openai/gpt-4o``. A prefix is only ever the
+    model's OWN provider: in an aggregator's route (``openrouter/openai/gpt-4o``)
+    the rest is that aggregator's model id, never the model a key for
+    ``openai/gpt-4o`` or ``gpt-4o`` names, so it is compared whole. An existing
+    instance of such a route records ``model="openai/gpt-4o"`` under
+    ``provider="openrouter"``, and is compared as ``openrouter/openai/gpt-4o``.
+    ``model:*`` matches every model. Nothing matches while the overlay is
+    building a model it mapped.
+
+    Args:
+        model: The model string an LLM is being built from, or the ``model``
+            of an existing instance.
+        provider: The provider the LLM routes to, when ``model`` does not say.
+
+    Returns:
+        The mapped model string, or ``None`` when no overlay is active, it has
+        no model key for ``model``, or a mapped model is being built.
+    """
+    mapping = active.get()
+    if not mapping or not model or _building_mapped.get():
+        return None
+    forms = _model_forms(model.strip(), provider)
+    for form in forms:
+        mapped = mapping.get(MODEL_KEY_PREFIX + form)
+        if mapped is not None:
+            return mapped
+    # The same model under another of its provider's names: a key written
+    # `google/x` for an llm that routed `gemini/x`, or the other way round.
+    aliases = _aliases()
+    canonical = {_canonical(form, aliases) for form in forms}
+    for key, mapped in mapping.items():
+        named = (
+            key[len(MODEL_KEY_PREFIX) :] if key.startswith(MODEL_KEY_PREFIX) else None
+        )
+        if named and named != ANY_MODEL and _canonical(named, aliases) in canonical:
+            return mapped
+    return mapping.get(MODEL_KEY_PREFIX + ANY_MODEL)
+
+
+def _model_forms(model: str, provider: str | None) -> list[str]:
+    """``model`` in the forms a key may name it by, the one as written first."""
+    if provider and not model.startswith(f"{provider}/"):
+        routed = f"{provider}/{model}"
+        # A bare name is the native one with its prefix stripped; a name that
+        # still has a slash is the provider's own route id and only means
+        # something beside the provider.
+        forms = [routed] if "/" in model else [model, routed]
+    else:
+        forms = [model]
+    prefix, separator, rest = forms[-1].partition("/")
+    if separator and "/" not in rest and prefix.lower() in _aliases():
+        if rest not in forms:
+            forms.append(rest)
+    return forms
+
+
+def _aliases() -> dict[str, str]:
+    """The router's provider aliases (``crewai.llm.PROVIDER_ALIASES``): every
+    prefix ``LLM`` strips from a model string it routes natively, to the
+    provider it names."""
+    from crewai.llm import PROVIDER_ALIASES
+
+    return PROVIDER_ALIASES
+
+
+def _canonical(form: str, aliases: dict[str, str]) -> str:
+    """``form`` with its provider prefix under the provider's one name."""
+    prefix, separator, rest = form.partition("/")
+    provider = aliases.get(prefix.lower()) if separator else None
+    return f"{provider}/{rest}" if provider else form
+
+
+def overlay_model_for_llm(llm: Any) -> str | None:
+    """The model the active overlay's model keys assign to an existing ``llm``.
+
+    An agent's declared ``llm`` built outside the block is looked up by its
+    ``model`` and ``provider``; one a model key already mapped when it was
+    built is not looked up again.
+    """
+    if llm is None or _mapped_llms.get(id(llm)) is llm:
+        return None
+    return overlay_model_for_model(
+        getattr(llm, "model", None), getattr(llm, "provider", None)
+    )
+
+
+def declared_before_overlay(llm: Any) -> Declared | None:
+    """What the caller declared for ``llm`` before the overlay mapped it, or None."""
+    if llm is None or _mapped_llms.get(id(llm)) is not llm:
+        return None
+    return _declared_before.get(id(llm))
+
+
+@contextmanager
+def building_mapped_model() -> Iterator[None]:
+    """Build the model the overlay mapped to, without mapping it again."""
+    token = _building_mapped.set(True)
+    try:
+        yield
+    finally:
+        _building_mapped.reset(token)
+
+
+def mark_mapped(llm: _T, declared: Declared | None = None) -> _T:
+    """Record ``llm`` as built by the overlay, so no model key maps it again.
+
+    ``declared`` is what the caller declared before the overlay mapped it —
+    the declared instance, or ``LLM(model, **kwargs)``'s arguments as
+    ``(model, kwargs, is_litellm)`` — so a role key can still be built from
+    the caller's own declaration (:func:`declared_before_overlay`).
+    """
+    try:
+        _mapped_llms[id(llm)] = llm
+        if declared is not None:
+            _declared_before[id(llm)] = declared
+            weakref.finalize(llm, _declared_before.pop, id(llm), None)
+    except TypeError:
+        # Only a weakly-referenceable object can be remembered (every
+        # ``BaseLLM`` is). Anything else is returned as built: the one cost is
+        # that an agent declared with it looks its model up again, as it would
+        # for an llm built outside the block — never a failed build.
+        pass
+    return llm
 
 
 def _stripped(mapping: dict[str, str] | None) -> dict[str, str] | None:
     """A copy of ``mapping`` with the whitespace around each role dropped.
 
-    Two keys that differ only by whitespace name one role. When they name the
-    same model the copy holds it once; when they name different models the
-    mapping is ambiguous and is refused, so the model an agent runs on never
-    depends on dictionary order.
+    Two keys that differ only by whitespace name one role (or, after the
+    :data:`MODEL_KEY_PREFIX`, one model). When they name the same model the
+    copy holds it once; when they name different models the mapping is
+    ambiguous and is refused, so the model an agent runs on never depends on
+    dictionary order.
     """
     if mapping is None:
         return None
     stripped: dict[str, str] = {}
     for role, model in mapping.items():
         key = role.strip()
+        if key.startswith(MODEL_KEY_PREFIX):
+            named = key[len(MODEL_KEY_PREFIX) :].strip()
+            if not named:
+                raise ValueError(
+                    f"llm_overlay: the key {role!r} names no model; write "
+                    f"'{MODEL_KEY_PREFIX}<provider/model>' or "
+                    f"'{MODEL_KEY_PREFIX}{ANY_MODEL}'"
+                )
+            key = MODEL_KEY_PREFIX + named
         if key in stripped and stripped[key] != model:
             raise ValueError(
                 f"llm_overlay: role {key!r} is mapped twice with different models "

lib/crewai/src/crewai/llms/context_window.py

@@ -0,0 +1,348 @@
+"""Shared context-window definitions and lookup for CrewAI LLMs."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from typing import Final
+
+
+MIN_CONTEXT: Final[int] = 1024
+MAX_CONTEXT: Final[int] = 10_000_000
+DEFAULT_CONTEXT_WINDOW_SIZE: Final[int] = 8192
+CONTEXT_WINDOW_USAGE_RATIO: Final[float] = 0.85
+
+# Raw provider limits. ``resolve_context_window_size`` applies the usable
+# context ratio uniformly, so all call sites share one matching rule.
+# Keep active IDs and context windows in sync with OpenAI's model catalog;
+# remove or replace entries when the vendor marks them retired:
+# https://developers.openai.com/api/docs/models
+OPENAI_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "gpt-4.1-mini-2025-04-14": 1_047_576,
+    "gpt-4.1-nano-2025-04-14": 1_047_576,
+    "gpt-6": 1_050_000,
+    "gpt-5.5": 1_050_000,
+    "gpt-5.4": 1_050_000,
+    "gpt-5.4-mini": 400_000,
+    "gpt-5.4-nano": 400_000,
+    "gpt-5.3-codex": 400_000,
+    "gpt-4-turbo": 128_000,
+    "gpt-4o-mini": 128_000,
+    "gpt-5-mini": 400_000,
+    "gpt-5-nano": 400_000,
+    "gpt-5.6": 1_050_000,
+    "gpt-5.6-cyber": 400_000,
+    "gpt-3.5-turbo": 16_385,
+    "gpt-oss-120b": 131_072,
+    "gpt-oss-20b": 131_072,
+    "chat-latest": 400_000,
+    "gpt-daybreak-blue-latest": 1_050_000,
+    "gpt-daybreak-red-latest": 400_000,
+    "o1-preview": 128_000,
+    "o1-mini": 128_000,
+    "o1-pro": 200_000,
+    "o1": 200_000,
+    "o3": 200_000,
+    "o3-mini": 200_000,
+    "o4-mini": 200_000,
+    "gpt-4.1": 1_047_576,
+    "gpt-4o": 128_000,
+    "gpt-5": 400_000,
+    "gpt-4": 8192,
+}
+
+# Check Azure deployment IDs and context windows here; availability varies by
+# account and region. Remove or replace vendor-retired entries:
+# https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure
+AZURE_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "gpt-6-astra": 1_050_000,
+    "gpt-5.6-sol": 1_050_000,
+    "gpt-5.6-terra": 1_050_000,
+    "gpt-5.6-luna": 1_050_000,
+    "gpt-chat-latest": 400_000,
+    "gpt-5.4-pro": 1_050_000,
+    "gpt-5.2-codex": 400_000,
+    "gpt-5.2": 400_000,
+    "gpt-5.1-codex-mini": 400_000,
+    "gpt-5.1-codex-max": 400_000,
+    "gpt-5.1-codex": 400_000,
+    "gpt-5.1": 400_000,
+    "gpt-5-codex": 400_000,
+    "gpt-5-pro": 400_000,
+    "gpt-oss-120b": 131_072,
+    "gpt-oss-20b": 131_072,
+    "codex-mini": 200_000,
+    "o3-pro": 200_000,
+    "computer-use-preview": 8192,
+    "gpt-4": 128_000,
+    "text-embedding-3-large": 8192,
+    "text-embedding-3-small": 8192,
+    "text-embedding-ada-002": 8192,
+    "text-embedding": 8191,
+}
+
+AZURE_OPENAI_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    **OPENAI_CONTEXT_WINDOWS,
+    **AZURE_CONTEXT_WINDOWS,
+}
+
+# Check active Claude IDs and context windows here; review retirements before
+# removing legacy prefixes: https://docs.anthropic.com/en/docs/about-claude/models
+# https://docs.anthropic.com/en/docs/about-claude/model-deprecations
+ANTHROPIC_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "claude-fable-5-1": 1_000_000,
+    "claude-fable-5": 1_000_000,
+    "claude-mythos-5-1": 1_000_000,
+    "claude-mythos-5": 1_000_000,
+    "claude-opus-5-5": 1_000_000,
+    "claude-opus-5": 1_000_000,
+    "claude-sonnet-5": 1_000_000,
+    "claude-opus-4-8": 1_000_000,
+    "claude-opus-4-7": 1_000_000,
+    "claude-sonnet-4-6": 1_000_000,
+    "claude-opus-4-6": 1_000_000,
+    "claude-opus-4-5": 200_000,
+    "claude-sonnet-4-5": 200_000,
+    "claude-haiku-4-5": 200_000,
+}
+
+# Check Gemini model IDs and context windows here; remove or replace retired
+# entries listed in the catalog:
+# https://ai.google.dev/gemini-api/docs/models
+GEMINI_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "gemini-3.8-flash": 1_048_576,
+    "gemini-3.7-flash": 1_048_576,
+    "gemini-3.6-flash": 1_048_576,
+    "gemini-3.5-flash": 1_048_576,
+    "gemini-3.5-flash-lite": 1_048_576,
+    "gemini-2.5-flash": 1_048_576,
+    "gemini-2.5-flash-lite": 1_048_576,
+    "gemini-2.5-pro": 1_048_576,
+    "gemini-3.1-flash-lite": 1_048_576,
+    "gemini-3.1-pro-preview": 1_048_576,
+    "gemini-3-flash-preview": 1_048_576,
+    "gemma-3-27b": 128_000,
+    "gemma-3-12b": 128_000,
+    "gemma-3-4b": 128_000,
+    "gemma-3-1b": 32_000,
+}
+
+# Check Bedrock model IDs and each model's context window in Models at a glance.
+# Availability varies by account and region; remove or replace retired entries:
+# https://docs.aws.amazon.com/bedrock/latest/userguide/models.html
+_BEDROCK_BASE_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "amazon.nova-2-lite-v1:0": 1_000_000,
+    "amazon.nova-pro-v1:0": 300_000,
+    "amazon.nova-micro-v1:0": 128_000,
+    "amazon.nova-lite-v1:0": 300_000,
+    "ai21.jamba-1-5-large-v1:0": 256_000,
+    "ai21.jamba-1-5-mini-v1:0": 256_000,
+    "deepseek.r1-v1:0": 128_000,
+    "deepseek.v3-v1:0": 128_000,
+    "deepseek.v3.2": 164_000,
+    "google.gemma-3-12b-it": 128_000,
+    "google.gemma-3-27b-it": 128_000,
+    "google.gemma-3-4b-it": 128_000,
+    "google.gemma-4-26b-a4b": 256_000,
+    "google.gemma-4-31b": 256_000,
+    "google.gemma-4-e2b": 128_000,
+    "meta.llama3-1-405b-instruct-v1:0": 128_000,
+    "meta.llama3-3-70b-instruct-v1:0": 128_000,
+    "meta.llama3-1-70b-instruct-v1:0": 128_000,
+    "meta.llama3-1-8b-instruct-v1:0": 128_000,
+    "meta.llama3-70b-instruct-v1:0": 8000,
+    "meta.llama3-8b-instruct-v1:0": 8000,
+    "meta.llama3-2-11b-instruct-v1:0": 128_000,
+    "meta.llama3-2-3b-instruct-v1:0": 128_000,
+    "meta.llama3-2-90b-instruct-v1:0": 128_000,
+    "meta.llama3-2-1b-instruct-v1:0": 128_000,
+    "meta.llama4-maverick-17b-instruct-v1:0": 1_000_000,
+    "meta.llama4-scout-17b-instruct-v1:0": 10_000_000,
+    "minimax.minimax-m2": 1_000_000,
+    "minimax.minimax-m2.1": 196_000,
+    "minimax.minimax-m2.5": 196_000,
+    "mistral.devstral-2-123b": 256_000,
+    "mistral.magistral-small-2509": 128_000,
+    "mistral.ministral-3-14b-instruct": 128_000,
+    "mistral.ministral-3-8b-instruct": 128_000,
+    "mistral.ministral-3-3b-instruct": 128_000,
+    "mistral.mistral-7b-instruct-v0:2": 32_000,
+    "mistral.mistral-large-3-675b-instruct": 256_000,
+    "mistral.mistral-large-2402-v1:0": 32_000,
+    "mistral.mistral-small-2402-v1:0": 32_000,
+    "mistral.mixtral-8x7b-instruct-v0:1": 32_000,
+    "mistral.pixtral-large-2502-v1:0": 128_000,
+    "moonshot.kimi-k2-thinking": 256_000,
+    "moonshotai.kimi-k2.5": 256_000,
+    "moonshotai.kimi-k3": 1_000_000,
+    "nvidia.nemotron-nano-12b-v2": 128_000,
+    "nvidia.nemotron-nano-9b-v2": 128_000,
+    "nvidia.nemotron-nano-3-30b": 256_000,
+    "nvidia.nemotron-super-3-120b": 256_000,
+    "openai.gpt-5.4": 1_000_000,
+    "openai.gpt-5.5": 1_000_000,
+    "openai.gpt-5.6-cyber": 272_000,
+    "openai.gpt-5.6-luna": 1_000_000,
+    "openai.gpt-5.6-sol": 1_000_000,
+    "openai.gpt-5.6-terra": 1_000_000,
+    "openai.gpt-6-astra": 1_050_000,
+    "openai.gpt-6-luna": 1_050_000,
+    "openai.gpt-6-sol": 1_050_000,
+    "openai.gpt-daybreak-blue-5.6-sol": 1_000_000,
+    "openai.gpt-oss-20b-1:0": 128_000,
+    "openai.gpt-oss-120b-1:0": 128_000,
+    "openai.gpt-oss-safeguard-20b": 128_000,
+    "openai.gpt-oss-safeguard-120b": 128_000,
+    "qwen.qwen3-235b-a22b-2507-v1:0": 256_000,
+    "qwen.qwen3-32b-v1:0": 32_000,
+    "qwen.qwen3-coder-30b-a3b-v1:0": 256_000,
+    "qwen.qwen3-coder-480b-a35b-v1:0": 128_000,
+    "qwen.qwen3-coder-next": 256_000,
+    "qwen.qwen3-next-80b-a3b": 256_000,
+    "qwen.qwen3-vl-235b-a22b": 256_000,
+    "writer.palmyra-x4-v1:0": 128_000,
+    "writer.palmyra-x5-v1:0": 128_000,
+    "xai.grok-4.3": 1_000_000,
+    "xai.grok-4.6": 500_000,
+    "zai.glm-4.7": 203_000,
+    "zai.glm-4.7-flash": 203_000,
+    "zai.glm-5": 200_000,
+}
+
+
+def _prefixed_context_windows(
+    sizes: Mapping[str, int], prefixes: Sequence[str]
+) -> dict[str, int]:
+    return {
+        f"{prefix}{model}": size for prefix in prefixes for model, size in sizes.items()
+    }
+
+
+# Bedrock's lifecycle is independent from Anthropic's direct API. Sonnet 4 is
+# Legacy (not EOL) on Bedrock, so keep its provider-specific limit until AWS
+# retires it: https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html
+_BEDROCK_LEGACY_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "anthropic.claude-sonnet-4": 200_000,
+    "us.anthropic.claude-sonnet-4": 200_000,
+    "eu.anthropic.claude-sonnet-4": 200_000,
+    "apac.anthropic.claude-sonnet-4": 200_000,
+    "global.anthropic.claude-sonnet-4": 200_000,
+    **_prefixed_context_windows(
+        {
+            "anthropic.claude-opus-4-1": 200_000,
+            "anthropic.claude-opus-4": 200_000,
+            "anthropic.claude-haiku-4": 200_000,
+            "anthropic.claude-3-7-sonnet": 200_000,
+            "anthropic.claude-3-5-sonnet": 200_000,
+            "anthropic.claude-3-5-haiku": 200_000,
+            "anthropic.claude-3-opus": 200_000,
+            "anthropic.claude-3-sonnet": 200_000,
+            "anthropic.claude-3-haiku": 200_000,
+        },
+        ("", "us.", "eu.", "apac.", "global."),
+    ),
+}
+
+# LiteLLM maintains its provider-only model IDs and context windows here. Prefer
+# the vendor catalogs above when both sources define the same model:
+# https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
+LITELLM_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    "deepseek-chat": 128_000,
+    "claude-3-5-sonnet": 200_000,
+    "gemma2-9b-it": 8192,
+    "gemma-7b-it": 8192,
+    "llama3-groq-70b-8192-tool-use-preview": 8192,
+    "llama3-groq-8b-8192-tool-use-preview": 8192,
+    "llama-3.1-70b-versatile": 131_072,
+    "llama-3.1-8b-instant": 131_072,
+    "llama-3.2-1b-preview": 8192,
+    "llama-3.2-3b-preview": 8192,
+    "llama-3.2-11b-text-preview": 8192,
+    "llama-3.2-90b-text-preview": 8192,
+    "llama3-70b-8192": 8192,
+    "llama3-8b-8192": 8192,
+    "mixtral-8x7b-32768": 32_768,
+    "llama-3.3-70b-versatile": 128_000,
+    "llama-3.3-70b-instruct": 128_000,
+    "Meta-Llama-3.3-70B-Instruct": 131_072,
+    "QwQ-32B-Preview": 8192,
+    "Qwen2.5-72B-Instruct": 8192,
+    "Qwen2.5-Coder-32B-Instruct": 8192,
+    "Meta-Llama-3.1-405B-Instruct": 8192,
+    "Meta-Llama-3.1-70B-Instruct": 131_072,
+    "Meta-Llama-3.1-8B-Instruct": 131_072,
+    "Llama-3.2-90B-Vision-Instruct": 16_384,
+    "Llama-3.2-11B-Vision-Instruct": 16_384,
+    "Meta-Llama-3.2-3B-Instruct": 4096,
+    "Meta-Llama-3.2-1B-Instruct": 16_384,
+    "gemini/gemma-3-1b-it": 32_000,
+    "gemini/gemma-3-4b-it": 128_000,
+    "gemini/gemma-3-12b-it": 128_000,
+    "gemini/gemma-3-27b-it": 128_000,
+    "mistral-tiny": 32_768,
+    "mistral-small-latest": 32_768,
+    "mistral-medium-latest": 32_768,
+    "mistral-large-latest": 32_768,
+    "mistral-large-2407": 32_768,
+    "mistral-large-2402": 32_768,
+    "mistral/mistral-tiny": 32_768,
+    "mistral/mistral-small-latest": 32_768,
+    "mistral/mistral-medium-latest": 32_768,
+    "mistral/mistral-large-latest": 32_768,
+    "mistral/mistral-large-2407": 32_768,
+    "mistral/mistral-large-2402": 32_768,
+}
+
+
+BEDROCK_CONTEXT_WINDOWS: Final[dict[str, int]] = {
+    **_BEDROCK_BASE_CONTEXT_WINDOWS,
+    **_BEDROCK_LEGACY_CONTEXT_WINDOWS,
+    **_prefixed_context_windows(_BEDROCK_BASE_CONTEXT_WINDOWS, ("us.", "eu.", "apac.")),
+    **_prefixed_context_windows(
+        ANTHROPIC_CONTEXT_WINDOWS,
+        (
+            "anthropic.",
+            "us.anthropic.",
+            "eu.anthropic.",
+            "apac.anthropic.",
+            "global.anthropic.",
+        ),
+    ),
+    # Geo inference ID is not produced by the us/eu/apac expansion.
+    "global.moonshotai.kimi-k3": 1_000_000,
+}
+
+LLM_CONTEXT_WINDOW_SIZES: Final[dict[str, int]] = {
+    **LITELLM_CONTEXT_WINDOWS,
+    **OPENAI_CONTEXT_WINDOWS,
+    **AZURE_CONTEXT_WINDOWS,
+    **ANTHROPIC_CONTEXT_WINDOWS,
+    **GEMINI_CONTEXT_WINDOWS,
+    **BEDROCK_CONTEXT_WINDOWS,
+}
+
+
+def resolve_context_window_size(
+    model: str,
+    sizes: Mapping[str, int],
+    *,
+    default: int,
+    extra_names: Sequence[str] = (),
+) -> int:
+    """Return the usable context window for a model using its longest prefix."""
+    for name, size in sizes.items():
+        if size < MIN_CONTEXT or size > MAX_CONTEXT:
+            raise ValueError(
+                f"Context window for {name} must be between {MIN_CONTEXT} and {MAX_CONTEXT}"
+            )
+
+    candidates = (model, *extra_names)
+    _, size = max(
+        (
+            (prefix, size)
+            for prefix, size in sizes.items()
+            if any(name.startswith(prefix) for name in candidates)
+        ),
+        key=lambda match: len(match[0]),
+        default=("", default),
+    )
+    return int(size * CONTEXT_WINDOW_USAGE_RATIO)

lib/crewai/src/crewai/llms/providers/anthropic/completion.py

@@ -1972,28 +1972,14 @@ class AnthropicCompletion(BaseLLM):
 
     def get_context_window_size(self) -> int:
         """Get the context window size for the model."""
-        from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO
+        from crewai.llms.context_window import (
+            ANTHROPIC_CONTEXT_WINDOWS,
+            resolve_context_window_size,
+        )
 
-        # Current offered models. Unknown and retired IDs fall back to 200k.
-        context_windows = {
-            "claude-fable-5": 1000000,
-            "claude-mythos-5": 1000000,
-            "claude-opus-5": 1000000,
-            "claude-sonnet-5": 1000000,
-            "claude-opus-4-8": 1000000,
-            "claude-opus-4-7": 1000000,
-            "claude-opus-4-6": 1000000,
-            "claude-sonnet-4-6": 1000000,
-            "claude-opus-4-5": 200000,
-            "claude-sonnet-4-5": 200000,
-            "claude-haiku-4-5": 200000,
-        }
-
-        for model_prefix, size in context_windows.items():
-            if self.model.startswith(model_prefix):
-                return int(size * CONTEXT_WINDOW_USAGE_RATIO)
-
-        return int(200000 * CONTEXT_WINDOW_USAGE_RATIO)
+        return resolve_context_window_size(
+            self.model, ANTHROPIC_CONTEXT_WINDOWS, default=200_000
+        )
 
     @staticmethod
     def _extract_finish_reason_and_id(

lib/crewai/src/crewai/llms/providers/azure/completion.py

@@ -1308,36 +1308,17 @@ class AzureCompletion(BaseLLM):
 
     def get_context_window_size(self) -> int:
         """Get the context window size for the model."""
-        from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM_CONTEXT_WINDOW_SIZES
+        from crewai.llms.context_window import (
+            AZURE_OPENAI_CONTEXT_WINDOWS,
+            DEFAULT_CONTEXT_WINDOW_SIZE,
+            resolve_context_window_size,
+        )
 
-        min_context = 1024
-        max_context = 2097152
-
-        for key, value in LLM_CONTEXT_WINDOW_SIZES.items():
-            if value < min_context or value > max_context:
-                raise ValueError(
-                    f"Context window for {key} must be between {min_context} and {max_context}"
-                )
-
-        # Longest prefix first. Always insert new keys in that order so
-        # startswith prefers gpt-5.6 over gpt-5, gpt-4o-mini over gpt-4o, etc.
-        context_windows = {
-            "text-embedding": 8191,
-            "gpt-3.5-turbo": 16385,
-            "gpt-5.4-mini": 200000,
-            "gpt-35-turbo": 16385,
-            "gpt-4o-mini": 128000,
-            "gpt-4-turbo": 128000,
-            "gpt-5.6": 1050000,
-            "gpt-4o": 128000,
-            "gpt-4": 8192,
-        }
-
-        for model_prefix, size in context_windows.items():
-            if self.model.startswith(model_prefix):
-                return int(size * CONTEXT_WINDOW_USAGE_RATIO)
-
-        return int(8192 * CONTEXT_WINDOW_USAGE_RATIO)
+        return resolve_context_window_size(
+            self.model,
+            AZURE_OPENAI_CONTEXT_WINDOWS,
+            default=DEFAULT_CONTEXT_WINDOW_SIZE,
+        )
 
     def _effective_max_tokens(self) -> int | float | None:
         """Azure reasoning/newer chat models cap via ``max_completion_tokens``."""

lib/crewai/src/crewai/llms/providers/bedrock/completion.py

@@ -2135,33 +2135,15 @@ class BedrockCompletion(BaseLLM):
 
     def get_context_window_size(self) -> int:
         """Get the context window size for the model."""
-        from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO
+        from crewai.llms.context_window import (
+            BEDROCK_CONTEXT_WINDOWS,
+            DEFAULT_CONTEXT_WINDOW_SIZE,
+            resolve_context_window_size,
+        )
 
-        context_windows = {
-            "anthropic.claude-sonnet-4": 200000,
-            "anthropic.claude-opus-4": 200000,
-            "anthropic.claude-haiku-4": 200000,
-            "anthropic.claude-3-5-sonnet": 200000,
-            "anthropic.claude-3-5-haiku": 200000,
-            "anthropic.claude-3-opus": 200000,
-            "anthropic.claude-3-sonnet": 200000,
-            "anthropic.claude-3-haiku": 200000,
-            "anthropic.claude-3-7-sonnet": 200000,
-            "anthropic.claude-v2": 100000,
-            "amazon.titan-text-express": 8000,
-            "ai21.j2-ultra": 8192,
-            "cohere.command-text": 4096,
-            "meta.llama2-13b-chat": 4096,
-            "meta.llama2-70b-chat": 4096,
-            "meta.llama3-70b-instruct": 128000,
-            "deepseek.r1": 32768,
-        }
-
-        for model_prefix, size in context_windows.items():
-            if self.model.startswith(model_prefix):
-                return int(size * CONTEXT_WINDOW_USAGE_RATIO)
-
-        return int(8192 * CONTEXT_WINDOW_USAGE_RATIO)
+        return resolve_context_window_size(
+            self.model, BEDROCK_CONTEXT_WINDOWS, default=DEFAULT_CONTEXT_WINDOW_SIZE
+        )
 
     def supports_multimodal(self) -> bool:
         """Check if the model supports multimodal inputs.

lib/crewai/src/crewai/llms/providers/gemini/completion.py

@@ -1382,40 +1382,14 @@ class GeminiCompletion(BaseLLM):
 
     def get_context_window_size(self) -> int:
         """Get the context window size for the model."""
-        from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM_CONTEXT_WINDOW_SIZES
+        from crewai.llms.context_window import (
+            GEMINI_CONTEXT_WINDOWS,
+            resolve_context_window_size,
+        )
 
-        min_context = 1024
-        max_context = 2097152
-
-        for key, value in LLM_CONTEXT_WINDOW_SIZES.items():
-            if value < min_context or value > max_context:
-                raise ValueError(
-                    f"Context window for {key} must be between {min_context} and {max_context}"
-                )
-
-        context_windows = {
-            "gemini-3.8-flash": 1048576,  # 1M tokens
-            "gemini-3-pro-preview": 1048576,  # 1M tokens
-            "gemini-2.0-flash": 1048576,  # 1M tokens
-            "gemini-2.0-flash-thinking": 32768,
-            "gemini-2.0-flash-lite": 1048576,
-            "gemini-2.5-flash": 1048576,
-            "gemini-2.5-pro": 1048576,
-            "gemini-1.5-pro": 2097152,  # 2M tokens
-            "gemini-1.5-flash": 1048576,
-            "gemini-1.5-flash-8b": 1048576,
-            "gemini-1.0-pro": 32768,
-            "gemma-3-1b": 32000,
-            "gemma-3-4b": 128000,
-            "gemma-3-12b": 128000,
-            "gemma-3-27b": 128000,
-        }
-
-        for model_prefix, size in context_windows.items():
-            if self.model.startswith(model_prefix):
-                return int(size * CONTEXT_WINDOW_USAGE_RATIO)
-
-        return int(1048576 * CONTEXT_WINDOW_USAGE_RATIO)  # 1M tokens default
+        return resolve_context_window_size(
+            self.model, GEMINI_CONTEXT_WINDOWS, default=1_048_576
+        )
 
     def _effective_max_tokens(self) -> int | float | None:
         """Gemini caps generation via ``max_output_tokens``."""

lib/crewai/src/crewai/llms/providers/openai/completion.py

@@ -2968,43 +2968,15 @@ class OpenAICompletion(BaseLLM):
 
     def get_context_window_size(self) -> int:
         """Get the context window size for the model."""
-        from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM_CONTEXT_WINDOW_SIZES
+        from crewai.llms.context_window import (
+            DEFAULT_CONTEXT_WINDOW_SIZE,
+            OPENAI_CONTEXT_WINDOWS,
+            resolve_context_window_size,
+        )
 
-        min_context = 1024
-        max_context = 2097152
-
-        for key, value in LLM_CONTEXT_WINDOW_SIZES.items():
-            if value < min_context or value > max_context:
-                raise ValueError(
-                    f"Context window for {key} must be between {min_context} and {max_context}"
-                )
-
-        # Longest prefix first. Always insert new keys in that order so
-        # startswith prefers gpt-5.6 over gpt-5, gpt-4o-mini over gpt-4o, etc.
-        context_windows = {
-            "gpt-4.1-mini-2025-04-14": 1047576,
-            "gpt-4.1-nano-2025-04-14": 1047576,
-            "gpt-5.4-mini": 200000,
-            "gpt-4-turbo": 128000,
-            "gpt-4o-mini": 128000,
-            "gpt-5-mini": 1047576,
-            "gpt-5-nano": 1047576,
-            "o1-preview": 128000,
-            "gpt-5.6": 1050000,
-            "o1-mini": 128000,
-            "o3-mini": 200000,
-            "o4-mini": 200000,
-            "gpt-4.1": 1047576,
-            "gpt-4o": 128000,
-            "gpt-5": 1047576,
-            "gpt-4": 8192,
-        }
-
-        for model_prefix, size in context_windows.items():
-            if self.model.startswith(model_prefix):
-                return int(size * CONTEXT_WINDOW_USAGE_RATIO)
-
-        return int(8192 * CONTEXT_WINDOW_USAGE_RATIO)
+        return resolve_context_window_size(
+            self.model, OPENAI_CONTEXT_WINDOWS, default=DEFAULT_CONTEXT_WINDOW_SIZE
+        )
 
     def _effective_max_tokens(self) -> int | float | None:
         """Newer OpenAI chat models cap via ``max_completion_tokens``."""

lib/crewai/src/crewai/telemetry/telemetry.py

@@ -1293,6 +1293,11 @@ class Telemetry:
     # its use under somebody else's name.
     FEATURE_ATTRIBUTES: ClassVar[dict[str, frozenset[str]]] = {
         "cli_usage:eval": frozenset({"authenticated"}),
+        # The models compared: a model in crewAI's own catalog by name (anyone
+        # can read it off a price list), any other — a fine-tune, a deployment
+        # name, a self-hosted model — as `<provider>/other`; the CLI decides
+        # (`telemetry_model_name`). Never a run, an output or an organization.
+        "cli_usage:eval_models": frozenset({"authenticated", "models", "models_count"}),
     }
 
     def feature_usage_span(

lib/crewai/src/crewai/telemetry/tracing/gen_ai_shapes.py

@@ -17,16 +17,41 @@ Spec: https://opentelemetry.io/docs/specs/semconv/gen-ai/
 
 from __future__ import annotations
 
+from functools import lru_cache
 import json
+import logging
+import math
 import os
 from typing import Any
 
 from crewai.utilities.serialization import to_serializable
 
 
+logger = logging.getLogger(__name__)
+
+
 _MAX_DEPTH = 14
 
-DEFAULT_MAX_ATTR_BYTES = 32 * 1024
+DEFAULT_MAX_ATTR_BYTES = 384 * 1024
+"""Upper bound, in UTF-8 bytes, on one exported span attribute value.
+
+Tool results, task outputs, agent prompts and answers, and LLM messages are
+evidence: whoever reads the trace (a person in the trace viewer, an evaluator
+checking that a summary matches the data a tool returned) needs them whole, so
+the bound sits well above what a real run produces (a 300 KB tool result fits).
+It is set by Wharf, which refuses an OTLP request whose encoded body is over
+3,072,000 bytes, so seven attributes at this bound still fit one request with
+room for the rest of the span (a span that does not fit is cut further at
+export, see ``grants._fit_span``). A value over the bound is cut as little as possible and says so:
+``<attr>.truncated`` and ``<attr>.original_size_bytes`` ride next to it.
+
+It is also the ceiling: ``CREWAI_OTEL_MAX_ATTR_BYTES`` and the OpenTelemetry
+SDK's span attribute length limit (``OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT``,
+else ``OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT``) can lower it, never raise it; a
+higher ``CREWAI_OTEL_MAX_ATTR_BYTES`` is clamped to it with one warning. The
+SDK cuts without a marker, so a lower SDK limit lowers the bound and the SDK
+never has anything left to cut.
+"""
 _PLACEHOLDER_ROLE = "system"
 _TRUNCATION_LOOP_LIMIT = 8
 
@@ -351,7 +376,7 @@ def truncate_attr(
     if payload is None:
         return None, {}
 
-    cap = max_bytes if max_bytes is not None else _max_attr_bytes()
+    cap = max_bytes if max_bytes is not None else max_attr_bytes()
     original_size = _byte_len(payload)
     if original_size <= cap:
         return payload, {}
@@ -371,15 +396,111 @@ def truncate_attr(
     return _envelope(payload, original_size, cap), markers
 
 
-def _max_attr_bytes() -> int:
-    raw = os.environ.get("CREWAI_OTEL_MAX_ATTR_BYTES")
+def _positive_env_int(name: str) -> int | None:
+    raw = os.environ.get(name)
     if not raw:
-        return DEFAULT_MAX_ATTR_BYTES
+        return None
     try:
         value = int(raw)
     except ValueError:
-        return DEFAULT_MAX_ATTR_BYTES
-    return value if value > 0 else DEFAULT_MAX_ATTR_BYTES
+        return None
+    return value if value > 0 else None
+
+
+_UNLIMITED = object()
+
+
+def _sdk_env_limit(name: str) -> int | None | object:
+    """Read one SDK length limit the way the OpenTelemetry SDK does.
+
+    Absent → ``None`` (fall through to the next setting); empty → unlimited;
+    a non-negative integer → that limit, ``0`` included (the SDK then cuts
+    every string to nothing). A value the SDK would reject is ignored here.
+    """
+    if name not in os.environ:
+        return None
+    raw = os.environ[name].strip().lower()
+    if raw == "":
+        return _UNLIMITED
+    try:
+        value = int(raw)
+    except ValueError:
+        return None
+    return value if value >= 0 else None
+
+
+def _sdk_span_attribute_limit() -> int | None:
+    """The SDK's span attribute length limit; ``None`` when unlimited.
+
+    The span setting takes precedence over the general one, also when it is
+    explicitly empty (unlimited), as in ``SpanLimits``.
+    """
+    for name in (
+        "OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT",
+        "OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT",
+    ):
+        limit = _sdk_env_limit(name)
+        if limit is _UNLIMITED:
+            return None
+        if isinstance(limit, int):
+            return limit
+    return None
+
+
+def max_attr_bytes() -> int:
+    """The byte bound :func:`truncate_attr` applies when given none.
+
+    See :data:`DEFAULT_MAX_ATTR_BYTES`. The SDK's limit counts characters and a
+    character is at least one byte, so a value within this many bytes is
+    within the SDK's limit too.
+    """
+    cap = DEFAULT_MAX_ATTR_BYTES
+    configured = _positive_env_int("CREWAI_OTEL_MAX_ATTR_BYTES")
+    if configured is not None:
+        if configured > DEFAULT_MAX_ATTR_BYTES:
+            _warn_clamped(configured)
+        else:
+            cap = configured
+    sdk_limit = _sdk_span_attribute_limit()
+    return min(cap, sdk_limit) if sdk_limit is not None else cap
+
+
+@lru_cache(maxsize=8)
+def _warn_clamped(configured: int) -> None:
+    """Say once per configured value that it was clamped, and why."""
+    logger.warning(
+        "CREWAI_OTEL_MAX_ATTR_BYTES=%d is above the %d-byte ceiling; using %d. "
+        "Wharf refuses an OTLP request over 3,072,000 bytes, and a span with "
+        "several attributes above the ceiling would not fit one request.",
+        configured,
+        DEFAULT_MAX_ATTR_BYTES,
+        DEFAULT_MAX_ATTR_BYTES,
+    )
+
+
+def truncate_plain(
+    payload: str | None,
+    *,
+    attr: str,
+    max_bytes: int | None = None,
+) -> tuple[str | None, dict[str, Any]]:
+    """Bound a plain (non-GenAI) string attribute: keep its head, mark the cut.
+
+    No shape is assumed: a JSON array whose items carry ``role`` may be a user
+    list or a chat log a task produced, not a GenAI conversation, so it is
+    never rewritten the way :func:`truncate_attr` rewrites messages.
+    """
+    if payload is None:
+        return None, {}
+    cap = max_bytes if max_bytes is not None else max_attr_bytes()
+    encoded = payload.encode("utf-8")
+    if len(encoded) <= cap:
+        return payload, {}
+    markers = {
+        f"{attr}.truncated": True,
+        f"{attr}.original_size_bytes": len(encoded),
+    }
+    return encoded[:cap].decode("utf-8", errors="ignore"), markers
 
 
 def _byte_len(s: str) -> int:
@@ -446,7 +567,14 @@ def _shrink_text_until_fits(
         if target is None:
             break
         part, content = target
-        new_content = _trunc_text(content, _byte_len(content) // 2)
+        # Cut what is over and no more. JSON escaping makes each byte of the
+        # text weigh more once serialized, so scale the cut by that weight;
+        # a later pass takes whatever is still over.
+        overshoot = _byte_len(serialized) - cap
+        raw_bytes = _byte_len(content)
+        weight = _byte_len(json.dumps(content)) / max(1, raw_bytes)
+        cut = math.ceil(overshoot / weight) + 64
+        new_content = _trunc_text(content, raw_bytes - cut)
         if new_content == content:
             break
         part["content"] = new_content
@@ -473,9 +601,12 @@ def _largest_text_part(
 
 
 def _trunc_text(content: str, target_bytes: int) -> str:
+    """Keep the head and the tail of ``content`` in about ``target_bytes``."""
     encoded = content.encode("utf-8")
-    head_bytes = max(target_bytes // 2, 256)
-    tail_bytes = max(target_bytes // 4, 128)
+    head_bytes = max(target_bytes * 2 // 3, 256)
+    tail_bytes = max(target_bytes // 3, 128)
+    if head_bytes + tail_bytes >= len(encoded):
+        return content
     head = encoded[:head_bytes].decode("utf-8", errors="ignore")
     tail = encoded[-tail_bytes:].decode("utf-8", errors="ignore")
     omitted_kb = max(1, (len(encoded) - head_bytes - tail_bytes) // 1024)
@@ -489,9 +620,9 @@ def _envelope(payload: str, original_size: int, cap: int) -> str | None:
     Used when the structural strategy doesn't apply (non-message JSON,
     malformed JSON) or didn't fit (rare — message envelopes dominate).
     """
-    preview_bytes = min(cap // 4, 4 * 1024)
     encoded = payload.encode("utf-8")
-    preview = encoded[:preview_bytes].decode("utf-8", errors="ignore")
+    # As much of the payload as fits: the preview is what a reader still has.
+    preview = encoded[:cap].decode("utf-8", errors="ignore")
     while True:
         envelope = json.dumps(
             {
@@ -500,9 +631,13 @@ def _envelope(payload: str, original_size: int, cap: int) -> str | None:
                 "_preview": preview,
             }
         )
-        if _byte_len(envelope) <= cap:
+        overshoot = _byte_len(envelope) - cap
+        if overshoot <= 0:
             return envelope
         if not preview:
             return None
-        # JSON escaping can expand the preview, so measure the encoded result.
-        preview = preview[: len(preview) // 2]
+        # JSON escaping can expand the preview, so measure what each kept
+        # character weighs once encoded and drop just enough of them.
+        weight = _byte_len(json.dumps(preview)) / len(preview)
+        drop = math.ceil(overshoot / weight) + 8
+        preview = preview[: max(0, len(preview) - drop)]

lib/crewai/src/crewai/telemetry/tracing/grants.py

@@ -9,6 +9,7 @@ from ipaddress import ip_address
 import logging
 import os
 from threading import Lock
+from typing import Any
 from urllib.parse import urlsplit
 from uuid import UUID
 
@@ -27,7 +28,8 @@ from crewai.events.listeners.tracing.utils import (
     is_tui_mode,
     should_suppress_tracing_messages,
 )
-from crewai.telemetry.tracing import last_run
+from crewai.telemetry.telemetry import Telemetry
+from crewai.telemetry.tracing import gen_ai_shapes, last_run
 from crewai.telemetry.tracing.session import MAX_EXPORT_BATCH_SIZE, otlp_exporter
 
 
@@ -36,6 +38,75 @@ logger = logging.getLogger(__name__)
 MAX_EXPORT_BODY_BYTES = 3_072_000
 
 
+_FIT_ATTEMPTS = 16
+
+
+def _fit_span(span: ReadableSpan, size: int) -> ReadableSpan | None:
+    """``span`` with its largest string attributes cut until it fits a request.
+
+    Each attribute is bounded on its own, so only a span carrying many large
+    ones gets here. Cutting the largest by the overshoot, with the usual
+    ``<attr>.truncated`` / ``<attr>.original_size_bytes`` markers, keeps the
+    span (its place in the tree, its timing, its status) where dropping it
+    would lose all of it. ``None`` when its strings cannot make it fit — the
+    excess is in the resource, the scope or the events.
+    """
+    attributes = dict(span.attributes or {})
+    for _ in range(_FIT_ATTEMPTS):
+        overshoot = size - MAX_EXPORT_BODY_BYTES
+        if overshoot <= 0:
+            return _with_attributes(span, attributes)
+        key, value = max(
+            (
+                (k, v)
+                for k, v in attributes.items()
+                if isinstance(v, str) and not k.endswith(".truncated")
+            ),
+            key=lambda item: len(item[1].encode("utf-8")),
+            default=(None, None),
+        )
+        if key is None or value is None:
+            return None
+        length = len(value.encode("utf-8"))
+        target = max(0, length - overshoot - 1024)
+        if key.startswith("gen_ai."):
+            cut, markers = gen_ai_shapes.truncate_attr(
+                value, attr=key, max_bytes=target
+            )
+        else:
+            cut, markers = gen_ai_shapes.truncate_plain(
+                value, attr=key, max_bytes=target
+            )
+        if cut is None:
+            cut = ""
+        if cut == value:
+            return None
+        attributes[key] = cut
+        # A value already cut keeps the size it had before the first cut.
+        markers.pop(f"{key}.original_size_bytes", None)
+        attributes.setdefault(f"{key}.original_size_bytes", length)
+        attributes.update(markers)
+        size = encode_spans([_with_attributes(span, attributes)]).ByteSize()
+    return None
+
+
+def _with_attributes(span: ReadableSpan, attributes: dict[str, Any]) -> ReadableSpan:
+    return ReadableSpan(
+        name=span.name,
+        context=span.get_span_context(),
+        parent=span.parent,
+        resource=span.resource,
+        attributes=attributes,
+        events=span.events,
+        links=span.links,
+        kind=span.kind,
+        status=span.status,
+        start_time=span.start_time,
+        end_time=span.end_time,
+        instrumentation_scope=span.instrumentation_scope,
+    )
+
+
 class TraceGrantError(Exception):
     """AMP could not authorize tracing; never downgrade a supplied credential."""
 
@@ -46,14 +117,36 @@ class TraceGrantError(Exception):
 
 def tracing_credential() -> str | None:
     """Resolve an explicit PAT, integration credential, or saved CLI login."""
+    resolved = resolve_tracing_credential()
+    return resolved[1] if resolved else None
+
+
+def tracing_credential_source() -> str | None:
+    """Which credential ``tracing_credential`` sends: ``"pat"``
+    (``CREWAI_USER_PAT``), ``"integration"`` (the platform integration token)
+    or ``"login"`` (the saved ``crewai login``) — so a refusal can name the one
+    that failed instead of sending somebody to refresh another."""
+    resolved = resolve_tracing_credential()
+    return resolved[0] if resolved else None
+
+
+def resolve_tracing_credential() -> tuple[str, str] | None:
+    """The credential tracing sends and where it came from, as ``(source,
+    token)`` — read once, in the one order both functions above follow.
+
+    A caller that sends the token and may later explain a refusal keeps this
+    pair: resolving the source again after the request can name a credential
+    AMP never saw (the environment or the context may have changed meanwhile).
+    """
     if token := os.getenv("CREWAI_USER_PAT"):
-        return token
+        return "pat", token
     if token := get_platform_integration_token():
-        return token
+        return "integration", token
     try:
-        return get_auth_token()
+        token = get_auth_token()
     except AuthError:
         return None
+    return ("login", token) if token else None
 
 
 @dataclass(frozen=True)
@@ -242,6 +335,18 @@ class GrantSpanExporter(SpanExporter):
             size = encode_spans(batch).ByteSize()
             if size > MAX_EXPORT_BODY_BYTES:
                 if len(batch) == 1:
+                    fitted = _fit_span(batch[0], size)
+                    if fitted is not None:
+                        logger.warning(
+                            "Execution trace span %r encoded to %d bytes, over "
+                            "Wharf's %d-byte request limit; its largest "
+                            "attributes were cut, each marked <attr>.truncated",
+                            batch[0].name,
+                            size,
+                            MAX_EXPORT_BODY_BYTES,
+                        )
+                        pending.append([fitted])
+                        continue
                     logger.warning(
                         "Skipping execution trace span: encoded size %d exceeds "
                         "Wharf's %d-byte request limit",
@@ -293,15 +398,19 @@ class GrantSpanExporter(SpanExporter):
             self._recorded = True
             execution_uuid = self._grant.execution_uuid
             api = getattr(self._client, "_api", None)
+            tier = getattr(self._client, "_tier", None)
             last_run.record_last_run(
                 execution_id=execution_uuid,
-                tier=getattr(self._client, "_tier", None),
+                tier=tier,
                 started_at_ns=self._first_start_ns,
                 finished_at_ns=self._last_end_ns,
                 amp_base_url=getattr(api, "base_url", None),
                 trace_url=self._trace_url,
             )
             logger.debug("Traces exported for execution %s", execution_uuid)
+            # Counts that a trace reached AMP, never its contents. The legacy
+            # TraceBatchManager emits the same names for runs outside a kickoff.
+            Telemetry().feature_usage_span(f"tracing:{tier}_sent")
             self._show_trace_link()
 
     def _show_trace_link(self) -> None:

lib/crewai/src/crewai/telemetry/tracing/handlers.py

@@ -112,7 +112,7 @@ from crewai.events.types.skill_events import (
     SkillUsedEvent,
 )
 from crewai.tasks.output_format import OutputFormat
-from crewai.telemetry.tracing import semantic_conventions
+from crewai.telemetry.tracing import gen_ai_shapes, semantic_conventions
 from crewai.telemetry.tracing.context import (
     PendingSpanEnd,
     TelemetryExecutionContext,
@@ -589,9 +589,25 @@ def _task_output_format(task: Any, output: Any = None) -> str:
 
 
 def _set_span_attributes(span: Span, attributes: dict[str, Any]) -> None:
+    """Set ``attributes`` on ``span``, no string value over the export bound.
+
+    The ``gen_ai.*`` content attributes arrive already bounded (and marked),
+    message-aware, by ``semantic_conventions``; this catches every other
+    string — a task's ``crewai.task.output``, an MCP tool's ``crewai.mcp.tool_result``, a flow's
+    serialized state — with a plain cut (the head, up to the bound) so one
+    oversized value marks itself instead of making the whole span too large
+    for Wharf to accept, and is never reshaped as if it were a conversation.
+    """
     for key, value in attributes.items():
-        if value is not None:
-            span.set_attribute(key, value)
+        if value is None:
+            continue
+        if isinstance(value, str) and f"{key}.truncated" not in attributes:
+            value, markers = gen_ai_shapes.truncate_plain(value, attr=key)
+            for marker, marker_value in markers.items():
+                span.set_attribute(marker, marker_value)
+            if value is None:
+                continue
+        span.set_attribute(key, value)
 
 
 def _get_parent_context(

lib/crewai/src/crewai/utilities/agent_utils.py

@@ -10,7 +10,7 @@ from datetime import datetime
 import inspect
 import json
 import re
-from typing import TYPE_CHECKING, Any, Final, Literal, TypedDict
+from typing import TYPE_CHECKING, Any, Final, Literal, cast
 
 from crewai_core.printer import PRINTER, ColoredText, Printer
 from crewai_core.settings import Settings
@@ -128,16 +128,6 @@ def resolve_plus_response(response: Any) -> Any:
     return asyncio.run(await_response())
 
 
-class SummaryContent(TypedDict):
-    """Structure for summary content entries.
-
-    Attributes:
-        content: The summarized content.
-    """
-
-    content: str
-
-
 console = Console()
 
 _MULTIPLE_NEWLINES: Final[re.Pattern[str]] = re.compile(r"\n+")
@@ -840,18 +830,6 @@ def handle_context_length(
         )
 
 
-def _estimate_token_count(text: str) -> int:
-    """Estimate token count using a conservative cross-provider heuristic.
-
-    Args:
-        text: The text to estimate tokens for.
-
-    Returns:
-        Estimated token count (roughly 1 token per 4 characters).
-    """
-    return len(text) // 4
-
-
 def _content_parts_text(content: list[dict[str, Any]]) -> str:
     """Text carried by a multimodal content-part list.
 
@@ -889,192 +867,281 @@ def message_content_text(msg: LLMMessage) -> str:
     return str(content)
 
 
-def _split_text_by_token_limit(text: str, max_tokens: int) -> list[str]:
-    """Split text into parts each estimated to fit within max_tokens."""
-    if not text:
-        return []
-    if _estimate_token_count(text) <= max_tokens:
-        return [text]
+class SummarizeMessages:
+    """Compact a message list so it fits the model context window.
 
-    # Inverse of _estimate_token_count (len // 4): each slice is at most max_tokens.
-    max_chars = max(1, max_tokens * 4)
-    return [text[i : i + max_chars] for i in range(0, len(text), max_chars)]
+    Preserves system messages, splits at message boundaries, formats with
+    role labels, and writes one structured summary back onto ``messages``.
+    Files attached to user messages are merged onto that summary.
+    """
 
+    _CHARS_PER_TOKEN_LEVELS: Final[tuple[float, ...]] = (4.0, 3.0, 2.5)
 
-def _expand_oversized_message(msg: LLMMessage, max_tokens: int) -> list[LLMMessage]:
-    """Split a message whose content alone exceeds max_tokens into sub-messages."""
-    msg_text = message_content_text(msg)
-    if _estimate_token_count(msg_text) <= max_tokens:
-        return [msg]
+    def __init__(self) -> None:
+        self.messages: list[LLMMessage] = []
+        self.llm: LLM | BaseLLM | None = None
+        self.callbacks: list[TokenCalcHandler] = []
+        self.verbose = True
 
-    # Reserve budget for the [Part i/n] prefix added to each sub-message.
-    body_max_tokens = max(1, max_tokens - 5)
-    parts = _split_text_by_token_limit(msg_text, body_max_tokens)
-    total_parts = len(parts)
-    expanded: list[LLMMessage] = []
+    def summarize(
+        self,
+        messages: list[LLMMessage],
+        llm: LLM | BaseLLM,
+        callbacks: list[TokenCalcHandler],
+        verbose: bool = True,
+    ) -> None:
+        """Replace non-system messages with a single summary, in place."""
+        if llm is None:
+            raise RuntimeError("SummarizeMessages.summarize() must set an LLM first.")
 
-    for index, part in enumerate(parts, start=1):
-        part_content = (
-            f"[Part {index}/{total_parts}]\n{part}" if total_parts > 1 else part
+        self.llm = llm
+        self.callbacks = callbacks
+        self.verbose = verbose
+        self.messages = messages
+        preserved_files = self._collect_attached_files()
+        system_messages = [m for m in self.messages if m.get("role") == "system"]
+        work_messages = [m for m in self.messages if m.get("role") != "system"]
+        if not work_messages:
+            return
+
+        chunks = self._chunk_messages(
+            work_messages,
+            llm.get_context_window_size(),
+            char_level_index=0,
         )
-        expanded.append({**msg, "content": part_content})
+        summaries = self._get_summaries_for_chunks(chunks, char_level_index=0)
+        self._replace_history_with_summary(system_messages, summaries, preserved_files)
 
-    return expanded
+    def _get_summaries_for_chunks(
+        self, chunks: list[list[LLMMessage]], char_level_index: int
+    ) -> list[str]:
+        total = len(chunks)
+        if self.verbose and total <= 1:
+            for index in range(1, total + 1):
+                PRINTER.print(
+                    content=f"Summarizing {index}/{total}...",
+                    color="yellow",
+                )
+        if self.verbose and total > 1:
+            PRINTER.print(
+                content=f"Summarizing {total} chunks in parallel...",
+                color="yellow",
+            )
+        return self._summarize_all(chunks, char_level_index)
 
+    def _summarize_all(
+        self, chunks: list[list[LLMMessage]], char_level_index: int
+    ) -> list[str]:
+        """Run one coroutine per chunk and return the summaries in order."""
 
-def _normalize_messages_for_chunking(
-    messages: list[LLMMessage], max_tokens: int
-) -> list[LLMMessage]:
-    """Return non-system messages with oversized entries split to fit max_tokens."""
-    normalized: list[LLMMessage] = []
-    for msg in messages:
-        if msg.get("role") == "system":
-            continue
-        normalized.extend(_expand_oversized_message(msg, max_tokens))
-    return normalized
+        async def _summarize_one(chunk: list[LLMMessage], level_index: int) -> str:
+            """Summarize one chunk; only reachable from ``_summarize_all`` or itself."""
+            llm = cast("LLM | BaseLLM", self.llm)
 
-
-def _format_messages_for_summary(messages: list[LLMMessage]) -> str:
-    """Format messages with role labels for summarization.
-
-    Skips system messages. Handles None content, tool_calls, and
-    multimodal content blocks.
-
-    Args:
-        messages: List of messages to format.
-
-    Returns:
-        Role-labeled conversation text.
-    """
-    lines: list[str] = []
-    for msg in messages:
-        role = msg.get("role", "user")
-        if role == "system":
-            continue
-
-        content = msg.get("content")
-        if content is None:
-            tool_calls = msg.get("tool_calls")
-            if tool_calls:
-                tool_names = []
-                for tc in tool_calls:
-                    func = tc.get("function", {})
-                    name = (
-                        func.get("name", "unknown")
-                        if isinstance(func, dict)
-                        else "unknown"
+            try:
+                summary = str(
+                    await llm.acall(
+                        self._build_summary_prompt(chunk), callbacks=self.callbacks
                     )
-                    tool_names.append(name)
-                content = f"[Called tools: {', '.join(tool_names)}]"
+                )
+            except LLMContextLengthExceededError:
+                pass
+            except Exception as error:
+                if not is_context_length_exceeded(error):
+                    raise
             else:
-                content = ""
-        elif isinstance(content, list):
-            content = _content_parts_text(content)
+                match = re.search(r"<summary>(.*?)</summary>", summary, re.DOTALL)
+                if match:
+                    return match.group(1).strip()
+                return summary.strip()
 
-        if role == "assistant":
-            label = "[ASSISTANT]:"
-        elif role == "tool":
-            tool_name = msg.get("name", "unknown")
-            label = f"[TOOL_RESULT ({tool_name})]:"
-        else:
-            label = "[USER]:"
+            if level_index + 1 >= len(self._CHARS_PER_TOKEN_LEVELS):
+                raise LLMContextLengthExceededError(
+                    "Summarization chunk still exceeds the context window after "
+                    f"retries at {self._CHARS_PER_TOKEN_LEVELS} chars-per-token levels."
+                ) from None
 
-        lines.append(f"{label} {content}")
+            next_level = level_index + 1
+            if self.verbose:
+                chars_per_token = self._CHARS_PER_TOKEN_LEVELS[next_level]
+                PRINTER.print(
+                    content=(
+                        "Summarization chunk exceeded context window; retrying with "
+                        f"tighter token estimate (1 token per {chars_per_token} chars)."
+                    ),
+                    color="yellow",
+                )
 
-    return "\n\n".join(lines)
+            sub_chunks = self._chunk_messages(
+                chunk,
+                llm.get_context_window_size(),
+                char_level_index=next_level,
+            )
+            if not sub_chunks:
+                raise LLMContextLengthExceededError(
+                    "Summarization chunk could not be split further."
+                ) from None
 
+            if len(sub_chunks) == 1:
+                return await _summarize_one(sub_chunks[0], next_level)
 
-def _split_messages_into_chunks(
-    messages: list[LLMMessage], max_tokens: int
-) -> list[list[LLMMessage]]:
-    """Split messages into chunks at message boundaries.
+            parts = await asyncio.gather(
+                *[_summarize_one(sub_chunk, next_level) for sub_chunk in sub_chunks]
+            )
+            return "\n\n".join(parts)
 
-    Excludes system messages and expands oversized single messages before
-    chunking. Each chunk stays under max_tokens based on estimated token count.
+        async def _gather() -> list[str]:
+            coroutines = [_summarize_one(chunk, char_level_index) for chunk in chunks]
+            return list(await asyncio.gather(*coroutines))
 
-    Args:
-        messages: List of messages to split.
-        max_tokens: Maximum estimated tokens per chunk.
+        coro = _gather()
+        if is_inside_event_loop():
+            ctx = contextvars.copy_context()
+            with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
+                return pool.submit(ctx.run, asyncio.run, coro).result()
+        return asyncio.run(coro)
 
-    Returns:
-        List of message chunks.
-    """
-    normalized = _normalize_messages_for_chunking(messages, max_tokens)
-    if not normalized:
-        return []
-
-    chunks: list[list[LLMMessage]] = []
-    current_chunk: list[LLMMessage] = []
-    current_tokens = 0
-
-    for msg in normalized:
-        msg_tokens = _estimate_token_count(message_content_text(msg))
-
-        if current_chunk and (current_tokens + msg_tokens) > max_tokens:
-            chunks.append(current_chunk)
-            current_chunk = []
-            current_tokens = 0
-
-        current_chunk.append(msg)
-        current_tokens += msg_tokens
-
-    if current_chunk:
-        chunks.append(current_chunk)
-
-    return chunks
-
-
-def _extract_summary_tags(text: str) -> str:
-    """Extract content between <summary></summary> tags.
-
-    Falls back to the full text if no tags are found.
-
-    Args:
-        text: Text potentially containing summary tags.
-
-    Returns:
-        Extracted summary content, or full text if no tags found.
-    """
-    match = re.search(r"<summary>(.*?)</summary>", text, re.DOTALL)
-    if match:
-        return match.group(1).strip()
-    return text.strip()
-
-
-async def _asummarize_chunks(
-    chunks: list[list[LLMMessage]],
-    llm: LLM | BaseLLM,
-    callbacks: list[TokenCalcHandler],
-) -> list[SummaryContent]:
-    """Summarize multiple message chunks concurrently using asyncio.
-
-    Args:
-        chunks: List of message chunks to summarize.
-        llm: LLM instance (must support ``acall``).
-        callbacks: List of callbacks for the LLM.
-
-    Returns:
-        Ordered list of summary contents, one per chunk.
-    """
-
-    async def _summarize_one(chunk: list[LLMMessage]) -> SummaryContent:
-        conversation_text = _format_messages_for_summary(chunk)
-        summarization_messages = [
+    def _build_summary_prompt(self, chunk: list[LLMMessage]) -> list[LLMMessage]:
+        conversation = self._conversation_text(chunk)
+        return [
             format_message_for_llm(
                 I18N_DEFAULT.slice("summarizer_system_message"), role="system"
             ),
             format_message_for_llm(
                 I18N_DEFAULT.slice("summarize_instruction").format(
-                    conversation=conversation_text
+                    conversation=conversation
                 ),
             ),
         ]
-        summary = await llm.acall(summarization_messages, callbacks=callbacks)
-        extracted = _extract_summary_tags(str(summary))
-        return {"content": extracted}
 
-    results = await asyncio.gather(*[_summarize_one(chunk) for chunk in chunks])
-    return list(results)
+    def _collect_attached_files(self) -> dict[str, Any]:
+        preserved: dict[str, Any] = {}
+        for msg in self.messages:
+            if msg.get("role") == "user" and msg.get("files"):
+                preserved.update(msg["files"])
+        return preserved
+
+    def _replace_history_with_summary(
+        self,
+        system_messages: list[LLMMessage],
+        summaries: list[str],
+        preserved_files: dict[str, Any],
+    ) -> None:
+        merged = "\n\n".join(summaries)
+        summary_message = format_message_for_llm(
+            I18N_DEFAULT.slice("summary").format(merged_summary=merged)
+        )
+        if preserved_files:
+            summary_message["files"] = preserved_files
+
+        self.messages.clear()
+        self.messages.extend(system_messages)
+        self.messages.append(summary_message)
+
+    def _approx_tokens(self, text: str, chars_per_token: float = 4.0) -> int:
+        """Estimate token count from character length and a chars-per-token heuristic."""
+        if not text:
+            return 0
+        return int(len(text) / chars_per_token)
+
+    def _chars_per_token(self, char_level_index: int) -> float:
+        return self._CHARS_PER_TOKEN_LEVELS[char_level_index]
+
+    def _messages_ready_to_chunk(
+        self,
+        messages: list[LLMMessage],
+        max_tokens: int,
+        char_level_index: int = 0,
+    ) -> list[LLMMessage]:
+        """Drop system messages and split any entry that exceeds max_tokens."""
+        chars_per_token = self._chars_per_token(char_level_index)
+        ready: list[LLMMessage] = []
+        for msg in messages:
+            if msg.get("role") == "system":
+                continue
+
+            text = message_content_text(msg)
+            if not text or self._approx_tokens(text, chars_per_token) <= max_tokens:
+                ready.append(msg)
+                continue
+
+            part_prefix_tokens = 5
+            body_max_tokens = max(1, max_tokens - part_prefix_tokens)
+            max_chars = max(1, int(body_max_tokens * chars_per_token))
+            parts = [text[i : i + max_chars] for i in range(0, len(text), max_chars)]
+            total_parts = len(parts)
+            for index, part in enumerate(parts, start=1):
+                ready.append(
+                    {**msg, "content": f"[Part {index}/{total_parts}]\n{part}"}
+                )
+        return ready
+
+    def _conversation_text(self, messages: list[LLMMessage]) -> str:
+        """Format messages with role labels, skipping system messages."""
+        lines: list[str] = []
+        for msg in messages:
+            role = msg.get("role", "user")
+            if role == "system":
+                continue
+
+            if role == "assistant":
+                prefix = "[ASSISTANT]:"
+            elif role == "tool":
+                prefix = f"[TOOL_RESULT ({msg.get('name', 'unknown')})]:"
+            else:
+                prefix = "[USER]:"
+
+            content = msg.get("content")
+            if content is None:
+                tool_calls = msg.get("tool_calls") or []
+                names = []
+                for tool_call in tool_calls:
+                    func = tool_call.get("function", {})
+                    names.append(
+                        func.get("name", "unknown")
+                        if isinstance(func, dict)
+                        else "unknown"
+                    )
+                body = f"[Called tools: {', '.join(names)}]" if names else ""
+            elif isinstance(content, list):
+                body = _content_parts_text(content)
+            else:
+                body = str(content)
+
+            lines.append(f"{prefix} {body}")
+        return "\n\n".join(lines)
+
+    def _chunk_messages(
+        self,
+        messages: list[LLMMessage],
+        max_tokens: int,
+        char_level_index: int = 0,
+    ) -> list[list[LLMMessage]]:
+        """Split messages into chunks that stay under max_tokens."""
+        chars_per_token = self._chars_per_token(char_level_index)
+        normalized = self._messages_ready_to_chunk(
+            messages, max_tokens, char_level_index=char_level_index
+        )
+        if not normalized:
+            return []
+
+        chunks: list[list[LLMMessage]] = []
+        current_chunk: list[LLMMessage] = []
+        current_tokens = 0
+        for msg in normalized:
+            msg_tokens = self._approx_tokens(
+                message_content_text(msg), chars_per_token=chars_per_token
+            )
+            if current_chunk and (current_tokens + msg_tokens) > max_tokens:
+                chunks.append(current_chunk)
+                current_chunk = []
+                current_tokens = 0
+            current_chunk.append(msg)
+            current_tokens += msg_tokens
+
+        if current_chunk:
+            chunks.append(current_chunk)
+        return chunks
 
 
 def summarize_messages(
@@ -1098,69 +1165,9 @@ def summarize_messages(
         callbacks: List of callbacks for LLM
         verbose: Whether to print progress.
     """
-    preserved_files: dict[str, Any] = {}
-    for msg in messages:
-        if msg.get("role") == "user" and msg.get("files"):
-            preserved_files.update(msg["files"])
-
-    system_messages = [m for m in messages if m.get("role") == "system"]
-    non_system_messages = [m for m in messages if m.get("role") != "system"]
-
-    if not non_system_messages:
-        return
-
-    max_tokens = llm.get_context_window_size()
-    chunks = _split_messages_into_chunks(non_system_messages, max_tokens)
-
-    total_chunks = len(chunks)
-
-    if total_chunks <= 1:
-        summarized_contents: list[SummaryContent] = []
-        for idx, chunk in enumerate(chunks, 1):
-            if verbose:
-                PRINTER.print(
-                    content=f"Summarizing {idx}/{total_chunks}...",
-                    color="yellow",
-                )
-            conversation_text = _format_messages_for_summary(chunk)
-            summarization_messages = [
-                format_message_for_llm(
-                    I18N_DEFAULT.slice("summarizer_system_message"), role="system"
-                ),
-                format_message_for_llm(
-                    I18N_DEFAULT.slice("summarize_instruction").format(
-                        conversation=conversation_text
-                    ),
-                ),
-            ]
-            summary = llm.call(summarization_messages, callbacks=callbacks)
-            extracted = _extract_summary_tags(str(summary))
-            summarized_contents.append({"content": extracted})
-    else:
-        if verbose:
-            PRINTER.print(
-                content=f"Summarizing {total_chunks} chunks in parallel...",
-                color="yellow",
-            )
-        coro = _asummarize_chunks(chunks=chunks, llm=llm, callbacks=callbacks)
-        if is_inside_event_loop():
-            ctx = contextvars.copy_context()
-            with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
-                summarized_contents = pool.submit(ctx.run, asyncio.run, coro).result()
-        else:
-            summarized_contents = asyncio.run(coro)
-
-    merged_summary = "\n\n".join(content["content"] for content in summarized_contents)
-
-    messages.clear()
-    messages.extend(system_messages)
-
-    summary_message = format_message_for_llm(
-        I18N_DEFAULT.slice("summary").format(merged_summary=merged_summary)
+    SummarizeMessages().summarize(
+        messages=messages, llm=llm, callbacks=callbacks, verbose=verbose
     )
-    if preserved_files:
-        summary_message["files"] = preserved_files
-    messages.append(summary_message)
 
 
 def show_agent_logs(

lib/crewai/src/crewai/utilities/llm_utils.py

@@ -6,6 +6,13 @@ from pydantic import ValidationError
 
 from crewai.constants import DEFAULT_LLM_MODEL, ENV_VARS, LITELLM_PARAMS
 from crewai.llm import LLM
+from crewai.llm_overlay import (
+    building_mapped_model,
+    declared_before_overlay,
+    mark_mapped,
+    overlay_model_for,
+    overlay_model_for_llm,
+)
 from crewai.llms.base_llm import BaseLLM
 
 
@@ -208,22 +215,114 @@ def create_llm_like(model: str, base: BaseLLM | None) -> BaseLLM:
     Returns:
         A new instance for ``model``; never ``base`` itself.
     """
-    if not isinstance(base, BaseLLM):
-        return LLM(model=model)
+    with building_mapped_model():
+        if not isinstance(base, BaseLLM):
+            return mark_mapped(LLM(model=model))
 
-    # The declared endpoint decides where an unknown ``openai/`` model routes (a
-    # custom OpenAI-compatible endpoint), so the same-provider question is asked
-    # with it: a self-hosted model mapped to another model on the same endpoint
-    # must keep that endpoint.
-    route = LLM._resolve_route(model, _configured_settings(base, _ENDPOINT_NAMES))
-    carried = _configured_settings(base, GENERATION_SETTINGS)
-    if _same_provider(base, route):
-        carried.update(_configured_settings(base, PROVIDER_SETTINGS))
-    target = LLM._resolve_route(model, carried).native_class or LLM
-    if type(base) is LLM and base.is_litellm:
+        # The declared endpoint decides where an unknown ``openai/`` model routes (a
+        # custom OpenAI-compatible endpoint), so the same-provider question is asked
+        # with it: a self-hosted model mapped to another model on the same endpoint
+        # must keep that endpoint.
+        route = LLM._resolve_route(model, _configured_settings(base, _ENDPOINT_NAMES))
+        carried = _configured_settings(base, GENERATION_SETTINGS)
+        if _same_provider(base, route):
+            carried.update(_configured_settings(base, PROVIDER_SETTINGS))
         # The declared llm runs through LiteLLM — by the caller's choice or
         # because no native class knew its model; either way that is the
         # environment its callbacks and extra kwargs were written for.
+        through_litellm = type(base) is LLM and base.is_litellm
+        return mark_mapped(
+            _build_like(model, carried, type(base), through_litellm), base
+        )
+
+
+def overlay_llm_for(role: str | None, declared: BaseLLM | None) -> BaseLLM | None:
+    """The llm an agent with ``role`` and ``declared`` llm runs on under ``llm_overlay``.
+
+    A role key wins, and its model is built from the caller's declaration even
+    when a model key already mapped ``declared`` on its way in; else a model
+    key matching ``declared``'s model (one a model key already mapped when it
+    was built is not looked up again); else ``declared`` itself. A mapped model
+    is built like ``declared`` (:func:`create_llm_like`). Outside any block
+    this is ``declared``.
+    """
+    role_model = overlay_model_for(role)
+    if role_model:
+        # The role wins, built from what the caller declared: when a model key
+        # already mapped the declared llm, its declaration, not the mapping.
+        before = declared_before_overlay(declared)
+        if isinstance(before, tuple):
+            declared_model, kwargs, is_litellm = before
+            return create_llm_from_kwargs_like(
+                role_model, declared_model, kwargs, is_litellm
+            )
+        base = before if isinstance(before, BaseLLM) else declared
+        return create_llm_like(role_model, base)
+    model = overlay_model_for_llm(declared)
+    return create_llm_like(model, declared) if model else declared
+
+
+def create_llm_from_kwargs_like(
+    model: str, declared_model: str, kwargs: dict[str, Any], is_litellm: bool
+) -> BaseLLM:
+    """Build ``model`` configured like ``LLM(declared_model, **kwargs)`` would be.
+
+    What ``llm_overlay``'s model keys build when ``LLM(model=...)`` is called
+    inside the block: the settings the caller passed are carried by the rule of
+    :func:`create_llm_like` — generation settings to any class that has them,
+    credentials and endpoints only to the same provider — without building the
+    declared model first (its provider's key may not be in this environment).
+    Everything the caller passed is theirs, so nothing counts as derived.
+    """
+    declaration = (declared_model, dict(kwargs), is_litellm)
+    with building_mapped_model():
+        settings = {
+            k: v
+            for k, v in kwargs.items()
+            if v is not None and not (isinstance(v, (list, dict)) and not v)
+        }
+        if isinstance(settings.get("endpoint"), str):
+            settings["endpoint"] = settings["endpoint"].split(_AZURE_DEPLOYMENT_PATH)[0]
+        endpoint = {k: settings[k] for k in _ENDPOINT_NAMES if k in settings}
+        route = LLM._resolve_route(model, endpoint)
+        carried = {k: settings[k] for k in GENERATION_SETTINGS if k in settings}
+        try:
+            declared = LLM._resolve_route(declared_model, kwargs)
+        except ImportError:
+            # The declared model's SDK is not installed here. Nothing is built
+            # on it, so that is no reason to fail; its credentials, though,
+            # cannot be matched to the new provider, so none are carried. A
+            # caller who chose LiteLLM keeps it.
+            return mark_mapped(
+                _build_like(model, carried, None, is_litellm), declaration
+            )
+        declared_class = (
+            LLM
+            if is_litellm or declared.native_class is None
+            else declared.native_class
+        )
+        if _same_route_provider(declared_class, declared.provider, route):
+            carried.update({k: settings[k] for k in PROVIDER_SETTINGS if k in settings})
+        return mark_mapped(
+            _build_like(model, carried, declared_class, declared_class is LLM),
+            declaration,
+        )
+
+
+def _build_like(
+    model: str,
+    carried: dict[str, Any],
+    base_class: type[BaseLLM] | None,
+    through_litellm: bool,
+) -> BaseLLM:
+    """Build ``model`` with the ``carried`` settings its class accepts.
+
+    ``base_class`` is the declared llm's class, ``None`` when it is not known;
+    ``additional_params`` travel only within one class.
+    """
+    route = LLM._resolve_route(model, carried)
+    target = route.native_class or LLM
+    if through_litellm:
         target = LLM
         carried["is_litellm"] = True
     accepted = {
@@ -231,7 +330,7 @@ def create_llm_like(model: str, base: BaseLLM | None) -> BaseLLM:
         for k, v in carried.items()
         if k in target.model_fields or k == "is_litellm"
     }
-    if type(base) is not target:
+    if base_class is not target:
         accepted.pop("additional_params", None)
     _carry_cap_under_the_targets_name(carried, accepted, target)
     if route.provider == "anthropic" and "temperature" in accepted:
@@ -239,6 +338,34 @@ def create_llm_like(model: str, base: BaseLLM | None) -> BaseLLM:
     return _build(model, accepted)
 
 
+def _same_route_provider(
+    declared_class: type[BaseLLM], declared_provider: str, route: Any
+) -> bool:
+    """:func:`_same_provider` for a declared llm known by its route, not an instance.
+
+    The same rule, case for case: a native declared class compares by class, a
+    LiteLLM-routed ``LLM`` by its provider — to a native route through the class
+    that provider names, so ``LLM(model="openai/gpt-4o", is_litellm=True)``
+    mapped to a native OpenAI model keeps its key and endpoint.
+    """
+    if route.native_class is None:
+        return declared_class is LLM and route.provider == declared_provider
+    if declared_class is route.native_class:
+        return not _serves_several_providers(route.native_class) or (
+            route.provider == declared_provider
+        )
+    if declared_class is not LLM:
+        return False
+    try:
+        named = LLM._get_native_provider(declared_provider or "")
+    except ImportError:
+        return False
+    return named is route.native_class and (
+        not _serves_several_providers(route.native_class)
+        or route.provider == declared_provider
+    )
+
+
 def _same_provider(base: BaseLLM, route: Any) -> bool:
     """Whether ``route`` lands where ``base``'s credentials and endpoint belong.
 

lib/crewai/tests/llms/anthropic/test_anthropic.py

@@ -5,7 +5,8 @@ from unittest.mock import AsyncMock, patch, MagicMock
 import pytest
 from pydantic import BaseModel
 
-from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM
+from crewai.llm import LLM
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.crew import Crew
 from crewai.agent import Agent
 from crewai.task import Task

lib/crewai/tests/llms/azure/test_azure.py

@@ -5,7 +5,8 @@ from unittest.mock import patch, MagicMock, Mock
 from urllib.parse import urlparse
 import pytest
 
-from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM
+from crewai.llm import LLM
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.crew import Crew
 from crewai.agent import Agent
 from crewai.task import Task
@@ -644,13 +645,33 @@ def test_azure_context_window_size():
     """
     Test that Azure models return correct context window sizes
     """
+    usable = int(128_000 * CONTEXT_WINDOW_USAGE_RATIO)
+
     llm_gpt4 = LLM(model="azure/gpt-4")
-    context_size_gpt4 = llm_gpt4.get_context_window_size()
-    assert context_size_gpt4 > 0
+    assert llm_gpt4.get_context_window_size() == usable
 
     llm_gpt4o = LLM(model="azure/gpt-4o")
-    context_size_gpt4o = llm_gpt4o.get_context_window_size()
-    assert context_size_gpt4o > context_size_gpt4  # GPT-4o has larger context
+    assert llm_gpt4o.get_context_window_size() == usable
+
+
+@pytest.mark.parametrize(
+    ("model", "context_window"),
+    [
+        ("azure/gpt-6-astra", 1_050_000),
+        ("azure/gpt-chat-latest", 400_000),
+        ("azure/gpt-oss-120b", 131_072),
+        ("azure/codex-mini", 200_000),
+        ("azure/computer-use-preview", 8192),
+        ("azure/gpt-4", 128_000),
+    ],
+)
+def test_azure_specific_models_use_official_context_windows(
+    model: str, context_window: int
+) -> None:
+    llm = LLM(model=model)
+    assert llm.get_context_window_size() == int(
+        context_window * CONTEXT_WINDOW_USAGE_RATIO
+    )
 
 
 @pytest.mark.parametrize(
@@ -670,7 +691,7 @@ def test_azure_gpt56_family_uses_official_context_window(model: str) -> None:
 
 def test_azure_gpt54_mini_keeps_its_window() -> None:
     llm = LLM(model="azure/gpt-5.4-mini")
-    assert llm.get_context_window_size() == int(200000 * CONTEXT_WINDOW_USAGE_RATIO)
+    assert llm.get_context_window_size() == int(400_000 * CONTEXT_WINDOW_USAGE_RATIO)
 
 
 def test_azure_message_formatting():

lib/crewai/tests/llms/bedrock/test_bedrock.py

@@ -5,6 +5,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
 import pytest
 
 from crewai.llm import LLM
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.crew import Crew
 from crewai.agent import Agent
 from crewai.task import Task
@@ -543,15 +544,23 @@ def test_bedrock_context_window_size():
     """
     Test that Bedrock models return correct context window sizes
     """
-    llm_claude = LLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0")
-    context_size_claude = llm_claude.get_context_window_size()
-    assert context_size_claude > 150000  # Should be substantial (200K tokens with ratio)
+    llm_nova = LLM(model="bedrock/amazon.nova-2-lite-v1:0")
+    context_size_nova = llm_nova.get_context_window_size()
+    assert context_size_nova == int(1_000_000 * CONTEXT_WINDOW_USAGE_RATIO)
 
     llm_titan = LLM(model="bedrock/amazon.titan-text-express-v1")
     context_size_titan = llm_titan.get_context_window_size()
     assert context_size_titan > 5000
 
 
+def test_bedrock_claude_sonnet_46_uses_its_specific_context_window():
+    llm = LLM(model="bedrock/anthropic.claude-sonnet-4-6-v1:0")
+
+    assert llm.get_context_window_size() == int(
+        1_000_000 * CONTEXT_WINDOW_USAGE_RATIO
+    )
+
+
 def test_bedrock_message_formatting():
     """
     Test that messages are properly formatted for Bedrock Converse API

lib/crewai/tests/llms/google/test_google.py

@@ -485,20 +485,20 @@ def test_gemini_context_window_size():
     """
     Test that Gemini models return correct context window sizes
     """
-    # Test Gemini 2.0 Flash
-    llm_2_0 = LLM(model="google/gemini-2.0-flash-001")
-    context_size_2_0 = llm_2_0.get_context_window_size()
-    assert context_size_2_0 > 500000
+    # Test Gemini 2.5 Flash
+    llm_2_5 = LLM(model="google/gemini-2.5-flash")
+    context_size_2_5 = llm_2_5.get_context_window_size()
+    assert context_size_2_5 > 500000
 
     # Test Gemini 3.8 Flash
     llm_3_8 = LLM(model="google/gemini-3.8-flash")
     context_size_3_8 = llm_3_8.get_context_window_size()
     assert context_size_3_8 == 891289
 
-    # Test Gemini 1.5 Pro
-    llm_1_5 = LLM(model="google/gemini-1.5-pro")
-    context_size_1_5 = llm_1_5.get_context_window_size()
-    assert context_size_1_5 > 1000000
+    # Test Gemini 2.5 Pro
+    llm_2_5_pro = LLM(model="google/gemini-2.5-pro")
+    context_size_2_5_pro = llm_2_5_pro.get_context_window_size()
+    assert context_size_2_5_pro > 500000
 
 
 def test_gemini_message_formatting():

lib/crewai/tests/llms/openai/test_openai.py

@@ -7,7 +7,8 @@ from unittest.mock import patch, MagicMock
 import openai
 import pytest
 
-from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, LLM
+from crewai.llm import LLM
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.llms.providers.openai.completion import OpenAICompletion, ResponsesAPIResult
 from crewai.crew import Crew
 from crewai.agent import Agent
@@ -213,6 +214,15 @@ def test_openai_completion_module_is_imported(monkeypatch):
     """
     module_name = "crewai.llms.providers.openai.completion"
 
+    # Re-importing binds the fresh module on its package too, and monkeypatch
+    # only restores sys.modules: without also restoring the attribute, every
+    # later test in this worker sees two module objects for one name, and a
+    # `patch("crewai.llms.providers.openai.completion.X")` patches the one that
+    # `from ... import X` does not read.
+    import crewai.llms.providers.openai as openai_package
+
+    if hasattr(openai_package, "completion"):
+        monkeypatch.setattr(openai_package, "completion", openai_package.completion)
     monkeypatch.delitem(sys.modules, module_name, raising=False)
 
     LLM(model="gpt-4o")
@@ -1877,8 +1887,8 @@ def test_openai_prefixed_gpt56_luna_uses_official_context_window() -> None:
 def test_openai_gpt5_and_gpt54_mini_keep_their_windows() -> None:
     gpt5 = OpenAICompletion(model="gpt-5")
     gpt54_mini = OpenAICompletion(model="gpt-5.4-mini")
-    assert gpt5.get_context_window_size() == int(1_047_576 * CONTEXT_WINDOW_USAGE_RATIO)
-    assert gpt54_mini.get_context_window_size() == int(200000 * CONTEXT_WINDOW_USAGE_RATIO)
+    assert gpt5.get_context_window_size() == int(400_000 * CONTEXT_WINDOW_USAGE_RATIO)
+    assert gpt54_mini.get_context_window_size() == int(400_000 * CONTEXT_WINDOW_USAGE_RATIO)
 
 
 def test_openai_stop_words_still_applied_to_regular_responses():

lib/crewai/tests/llms/test_context_window.py

@@ -0,0 +1,116 @@
+import pytest
+
+from crewai.llms.context_window import (
+    BEDROCK_CONTEXT_WINDOWS,
+    GEMINI_CONTEXT_WINDOWS,
+    CONTEXT_WINDOW_USAGE_RATIO,
+    DEFAULT_CONTEXT_WINDOW_SIZE,
+    MAX_CONTEXT,
+    LLM_CONTEXT_WINDOW_SIZES,
+    resolve_context_window_size,
+)
+
+
+def test_resolver_prefers_the_longest_matching_prefix() -> None:
+    sizes = {"gpt-5": 1_047_576, "gpt-5.4-mini": 200_000}
+
+    result = resolve_context_window_size(
+        "gpt-5.4-mini-2026-01-01", sizes, default=DEFAULT_CONTEXT_WINDOW_SIZE
+    )
+
+    assert result == int(200_000 * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+def test_resolver_uses_default_for_an_unknown_model() -> None:
+    result = resolve_context_window_size(
+        "unknown-model", {}, default=DEFAULT_CONTEXT_WINDOW_SIZE
+    )
+
+    assert result == int(DEFAULT_CONTEXT_WINDOW_SIZE * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+def test_resolver_validates_all_declared_context_windows() -> None:
+    with pytest.raises(ValueError, match="must be between 1024 and 10000000"):
+        resolve_context_window_size("test-model", {"test-model": 500}, default=8192)
+
+
+def test_bedrock_claude_aliases_use_the_anthropic_context_window() -> None:
+    result = resolve_context_window_size(
+        "us.anthropic.claude-sonnet-4-6-v1:0",
+        BEDROCK_CONTEXT_WINDOWS,
+        default=DEFAULT_CONTEXT_WINDOW_SIZE,
+    )
+
+    assert result == int(1_000_000 * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+def test_bedrock_regional_aliases_preserve_the_base_model_context_window() -> None:
+    result = resolve_context_window_size(
+        "us.meta.llama3-3-70b-instruct-v1:0",
+        BEDROCK_CONTEXT_WINDOWS,
+        default=DEFAULT_CONTEXT_WINDOW_SIZE,
+    )
+
+    assert result == int(128_000 * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+def test_litellm_map_includes_the_openai_gpt5_family() -> None:
+    result = resolve_context_window_size(
+        "gpt-5", LLM_CONTEXT_WINDOW_SIZES, default=DEFAULT_CONTEXT_WINDOW_SIZE
+    )
+
+    assert result == int(400_000 * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+@pytest.mark.parametrize(
+    ("model", "raw_context_window"),
+    [
+        ("o1", 200_000),
+        ("o1-pro", 200_000),
+        ("o3", 200_000),
+        ("gemini-3.1-flash-lite", 1_048_576),
+        ("gemini-3.1-pro-preview", 1_048_576),
+        ("gemini-3-flash-preview", 1_048_576),
+    ],
+)
+def test_affected_model_ids_use_their_specific_context_windows(
+    model: str, raw_context_window: int
+) -> None:
+    assert resolve_context_window_size(
+        model, LLM_CONTEXT_WINDOW_SIZES, default=DEFAULT_CONTEXT_WINDOW_SIZE
+    ) == int(raw_context_window * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+@pytest.mark.parametrize(
+    ("model", "sizes", "raw_context_window"),
+    [
+        ("gpt-6-astra", LLM_CONTEXT_WINDOW_SIZES, 1_050_000),
+        ("gpt-5.6-cyber", LLM_CONTEXT_WINDOW_SIZES, 400_000),
+        ("gpt-3.5-turbo", LLM_CONTEXT_WINDOW_SIZES, 16_385),
+        ("chat-latest", LLM_CONTEXT_WINDOW_SIZES, 400_000),
+        ("gpt-oss-120b", LLM_CONTEXT_WINDOW_SIZES, 131_072),
+        ("gpt-5.4-nano", LLM_CONTEXT_WINDOW_SIZES, 400_000),
+        ("gemini-3.7-flash", LLM_CONTEXT_WINDOW_SIZES, 1_048_576),
+        ("gemma-3-1b-it", GEMINI_CONTEXT_WINDOWS, 32_000),
+        ("gemma-3-27b-it", GEMINI_CONTEXT_WINDOWS, 128_000),
+        ("amazon.nova-2-lite-v1:0", BEDROCK_CONTEXT_WINDOWS, 1_000_000),
+        ("global.moonshotai.kimi-k3", BEDROCK_CONTEXT_WINDOWS, 1_000_000),
+        (
+            "anthropic.claude-3-5-sonnet-20241022-v2:0",
+            BEDROCK_CONTEXT_WINDOWS,
+            200_000,
+        ),
+        ("us.anthropic.claude-opus-4-1-20250805-v1:0", BEDROCK_CONTEXT_WINDOWS, 200_000),
+        (
+            "meta.llama4-scout-17b-instruct-v1:0",
+            BEDROCK_CONTEXT_WINDOWS,
+            MAX_CONTEXT,
+        ),
+    ],
+)
+def test_new_catalog_models_use_their_documented_context_windows(
+    model: str, sizes: dict[str, int], raw_context_window: int
+) -> None:
+    assert resolve_context_window_size(
+        model, sizes, default=DEFAULT_CONTEXT_WINDOW_SIZE
+    ) == int(raw_context_window * CONTEXT_WINDOW_USAGE_RATIO)

lib/crewai/tests/telemetry/test_grant_export_bounds.py

@@ -113,17 +113,52 @@ def test_exact_body_limit_and_oversized_span_preserve_fitting_neighbors(
     assert encode_spans([large]).ByteSize() == BODY_LIMIT + extra_bytes
     spans = [make_span(0), large, make_span(2)]
 
-    expected = SpanExportResult.FAILURE if extra_bytes else SpanExportResult.SUCCESS
-    assert exporter.export(spans) == expected
+    # One byte over is no longer a dropped span: its largest attribute is cut,
+    # marked, and the span is sent with its neighbours.
+    assert exporter.export(spans) == SpanExportResult.SUCCESS
 
     exported = [span for batch in exported_batches(delegate) for span in batch]
-    assert exported == ([spans[0], spans[2]] if extra_bytes else spans)
+    assert [span.context.span_id for span in exported] == [1, 2, 3]
     if extra_bytes:
+        fitted = exported[1]
+        assert fitted.attributes["gen_ai.input.messages.truncated"] is True
+        assert fitted.attributes[
+            "gen_ai.input.messages.original_size_bytes"
+        ] == BODY_LIMIT - overhead + extra_bytes
+        assert encode_spans([fitted]).ByteSize() <= BODY_LIMIT
         assert "3072001" in caplog.text and "3072000" in caplog.text
         assert "synthetic-grant" not in caplog.text
+    else:
+        assert exported == spans
     client.create.assert_not_called()
 
 
+def test_a_span_with_many_large_attributes_is_shrunk_not_dropped(
+    destination, caplog
+):
+    """Each attribute fits its own bound, the span does not fit a request: the
+    largest are cut, each marked, until it fits; nothing else changes."""
+    exporter, _, delegate = destination
+    attributes = {f"crewai.part.{i}": f"{i}" * 390_000 for i in range(9)}
+    attributes["crewai.task.name"] = "summary"
+    span = make_span(0, attributes=attributes)
+    assert encode_spans([span]).ByteSize() > BODY_LIMIT
+
+    assert exporter.export([span]) == SpanExportResult.SUCCESS
+
+    (fitted,) = [s for batch in exported_batches(delegate) for s in batch]
+    assert fitted.context == span.context and fitted.name == span.name
+    assert fitted.attributes["crewai.task.name"] == "summary"
+    cut = [k for k in attributes if fitted.attributes.get(f"{k}.truncated")]
+    assert cut
+    for key in cut:
+        assert fitted.attributes[f"{key}.original_size_bytes"] == 390_000
+        assert attributes[key].startswith(fitted.attributes[key])
+    for key in set(attributes) - set(cut):
+        assert fitted.attributes[key] == attributes[key]
+    assert "were cut" in caplog.text
+
+
 def test_oversized_metadata_returns_failure_without_sending(destination, caplog):
     exporter, _, delegate = destination
     span = make_span(

lib/crewai/tests/telemetry/test_session_trace_export.py

@@ -711,9 +711,9 @@ def test_first_time_execution_uses_local_session_even_with_saved_credentials(
         "crewai.events.listeners.tracing.utils.should_auto_collect_first_time_traces",
         lambda: True,
     )
-    credential = Mock(return_value="saved-login")
+    credential = Mock(return_value=("login", "saved-login"))
     monkeypatch.setattr(
-        "crewai.telemetry.tracing.grants.tracing_credential", credential
+        "crewai.telemetry.tracing.grants.resolve_tracing_credential", credential
     )
     save = Mock()
     monkeypatch.setattr(ephemeral, "update_user_data", save)
@@ -847,3 +847,142 @@ def test_invalid_buffer_limits_use_safe_defaults(monkeypatch, invalid):
     buffer = EphemeralSpanBuffer()
     assert buffer._max_spans == 1000 and buffer._max_bytes == 8388608
     buffer.shutdown()
+
+
+@pytest.fixture
+def sent_features(monkeypatch):
+    """The `tracing:*` Feature Usage names emitted, in order."""
+    from crewai.telemetry.telemetry import Telemetry
+
+    features = []
+    monkeypatch.setattr(
+        Telemetry,
+        "feature_usage_span",
+        lambda self, feature: features.append(feature)
+        if feature.startswith("tracing:")
+        else None,
+    )
+    return features
+
+
+@pytest.mark.parametrize(
+    ("authenticated", "approved", "grant_status", "export_status", "expected"),
+    [
+        (True, True, 200, 200, ["tracing:authenticated_sent"]),
+        (False, True, 200, 200, ["tracing:ephemeral_sent"]),
+        (False, False, 200, 200, []),
+        (True, True, 200, 401, []),
+        (False, True, 200, 401, []),
+        (False, True, 500, 200, []),
+    ],
+)
+def test_a_trace_that_reaches_amp_is_counted_once_by_tier(
+    collector,
+    sent_features,
+    monkeypatch,
+    authenticated,
+    approved,
+    grant_status,
+    export_status,
+    expected,
+):
+    """Declined consent, a refused grant, or a rejected export never counts."""
+    from crewai.execution import begin_execution, end_execution
+    from crewai.telemetry.tracing.context import get_trace_session
+
+    collector.grant_status = grant_status
+    collector.export_status = export_status
+    if authenticated:
+        monkeypatch.setenv("CREWAI_USER_PAT", "synthetic-pat")
+
+    def run():
+        with trace_consent(lambda: approved):
+            token = begin_execution(tracing=True)
+            try:
+                record(get_trace_session())
+                nested = begin_execution(tracing=True)
+                end_execution(nested)
+            finally:
+                end_execution(token)
+
+    copy_context().run(run)
+    assert sent_features == expected
+
+
+def test_a_deferred_run_is_counted_once_when_it_finally_ends(
+    collector, sent_features, monkeypatch
+):
+    from crewai.execution import begin_execution, end_execution
+    from crewai.telemetry.tracing.context import get_trace_session
+
+    monkeypatch.setenv("CREWAI_USER_PAT", "synthetic-pat")
+    token = begin_execution(tracing=True)
+    try:
+        session = get_trace_session()
+        record(session)
+        session.flush()
+    finally:
+        lifetime = end_execution(token, defer=True)
+    assert sent_features == []  # spans reached AMP, but the run is not over
+
+    token = begin_execution(tracing=True, trace_session=lifetime)
+    try:
+        record(get_trace_session(), "second turn")
+    finally:
+        end_execution(token)
+    assert sent_features == ["tracing:authenticated_sent"]
+
+
+def test_recording_the_same_export_twice_counts_it_once(collector, sent_features):
+    client = TraceGrantClient(None)
+    grant = client.create(str(uuid4()))
+    exporter = GrantSpanExporter(client, grant)
+    session = TraceSession(grant.execution_uuid, [exporter])
+    try:
+        record(session)
+    finally:
+        session.shutdown()
+    exporter.record_export()
+    exporter.record_export()
+    assert sent_features == ["tracing:ephemeral_sent"]
+
+
+def test_a_truncated_ephemeral_trace_is_not_counted(
+    collector, sent_features, monkeypatch
+):
+    """It is uploaded, but like the `crewai eval` record it is not a whole run."""
+    monkeypatch.setenv("CREWAI_EPHEMERAL_TRACE_MAX_SPANS", "1")
+    with trace_consent(lambda: True), ephemeral_tracing(str(uuid4())) as session:
+        record(session, "first")
+        record(session, "second")
+    assert len(collector.batches) == 1
+    assert sent_features == []
+
+
+@pytest.mark.parametrize("authenticated", [True, False])
+@pytest.mark.parametrize("use_async", [False, True])
+def test_a_flow_kickoff_counts_its_trace(
+    collector, sent_features, monkeypatch, authenticated, use_async
+):
+    from crewai.flow.flow import Flow, start
+
+    if authenticated:
+        monkeypatch.setenv("CREWAI_USER_PAT", "synthetic-pat")
+
+    class Greeting(Flow):
+        @start()
+        def greet(self):
+            return "hello"
+
+    def run():
+        with trace_consent(lambda: True):
+            flow = Greeting(tracing=True)
+            if use_async:
+                return asyncio.run(flow.kickoff_async())
+            return flow.kickoff()
+
+    assert copy_context().run(run) == "hello"
+    assert collector.batches
+    assert sent_features == [
+        "tracing:authenticated_sent" if authenticated else "tracing:ephemeral_sent"
+    ]

lib/crewai/tests/telemetry/test_span_attribute_bound.py

@@ -0,0 +1,403 @@
+"""What a run produced reaches the exported span whole, up to one stated bound.
+
+A graded run of a flow could not be judged because the span carried a cut copy
+of a tool's result: a summary task had to report "the issue count and
+source_issue_ids exactly match the Linear results", and the grader saw 4 KB of
+a 33 KB result. Tool results, task outputs and LLM messages are what a reader
+of the trace checks the run against, so they now arrive whole up to
+``DEFAULT_MAX_ATTR_BYTES`` (sized to Wharf's request limit), and a value over
+it is cut as little as possible and says so — never silently.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+import json
+
+from crewai import Agent, Crew, Task
+from crewai.events.types.task_events import TaskCompletedEvent, TaskStartedEvent
+from crewai.events.types.tool_usage_events import (
+    ToolUsageFinishedEvent,
+    ToolUsageStartedEvent,
+)
+from crewai.tasks.task_output import TaskOutput
+from crewai.telemetry.tracing import gen_ai_shapes, handlers, semantic_conventions
+from crewai.telemetry.tracing.context import TelemetryExecutionContext
+from crewai.telemetry.tracing.grants import MAX_EXPORT_BODY_BYTES
+from opentelemetry.exporter.otlp.proto.common.trace_encoder import encode_spans
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
+import pytest
+
+
+BOUND = gen_ai_shapes.DEFAULT_MAX_ATTR_BYTES
+
+
+def _text(size: int, label: str = "issue") -> str:
+    """Non-repeating text of about ``size`` bytes: a cut copy cannot compare equal."""
+    lines: list[str] = []
+    total = 0
+    i = 0
+    while total < size:
+        line = f'{{"id": "{label}-{i}", "title": "Linear issue number {i}"}}\n'
+        lines.append(line)
+        total += len(line)
+        i += 1
+    return "".join(lines)
+
+
+@pytest.fixture(autouse=True)
+def enable_otel_sdk(monkeypatch: pytest.MonkeyPatch) -> None:
+    """The suite otherwise runs with OTEL_SDK_DISABLED, which makes every
+    assertion here pass vacuously against non-recording spans."""
+    for name in (
+        "OTEL_SDK_DISABLED",
+        "CREWAI_DISABLE_TELEMETRY",
+        "CREWAI_DISABLE_TRACKING",
+        "CREWAI_OTEL_MAX_ATTR_BYTES",
+        "OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT",
+        "OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT",
+    ):
+        monkeypatch.delenv(name, raising=False)
+
+
+class _Providers:
+    def __init__(self, tracer) -> None:
+        self._tracer = tracer
+
+    def get_tracer(self, name: str | None = None):
+        return self._tracer
+
+    def emit_log(self, *args, **kwargs) -> None:
+        pass
+
+
+def _pipeline():
+    """Built inside each test, after any env change: the SDK reads its span
+    limits when the provider is constructed."""
+    exporter = InMemorySpanExporter()
+    provider = TracerProvider()
+    provider.add_span_processor(SimpleSpanProcessor(exporter))
+    tracer = provider.get_tracer("test")
+    ctx = TelemetryExecutionContext(
+        kickoff_id="kickoff", automation_name="test", tracer=tracer
+    )
+    return provider, _Providers(tracer), ctx, exporter
+
+
+def _only_span(exporter: InMemorySpanExporter, name: str):
+    matches = [s for s in exporter.get_finished_spans() if s.name == name]
+    assert len(matches) == 1, [s.name for s in exporter.get_finished_spans()]
+    return matches[0]
+
+
+def _tool_span(result: str):
+    provider, providers, ctx, exporter = _pipeline()
+    now = datetime.now(timezone.utc)
+    args = {"query": "issues in cycle 42"}
+    started = ToolUsageStartedEvent(
+        tool_name="linear_run_query", tool_args=args, agent_key="k", agent_role="r"
+    )
+    handlers.handle_tool_usage_started(providers, ctx, None, started)
+    handlers.handle_tool_usage_finished(
+        providers,
+        ctx,
+        None,
+        ToolUsageFinishedEvent(
+            tool_name="linear_run_query",
+            tool_args=args,
+            agent_key="k",
+            agent_role="r",
+            started_at=now,
+            finished_at=now,
+            output=result,
+            started_event_id=started.event_id,
+        ),
+    )
+    span = _only_span(exporter, "call tool")
+    provider.shutdown()
+    return span
+
+
+@pytest.mark.parametrize("size", [50_000, 300_000], ids=["50KB", "300KB"])
+def test_a_tool_result_arrives_whole(size: int) -> None:
+    result = _text(size)
+
+    span = _tool_span(result)
+
+    assert json.loads(span.attributes["gen_ai.tool.call.result"]) == result
+    assert "gen_ai.tool.call.result.truncated" not in span.attributes
+
+
+def test_a_tool_result_over_the_bound_is_cut_with_the_marker_and_its_size() -> None:
+    result = _text(BOUND + 200_000)
+    original = len(json.dumps(result).encode("utf-8"))
+
+    span = _tool_span(result)
+
+    payload = span.attributes["gen_ai.tool.call.result"]
+    assert span.attributes["gen_ai.tool.call.result.truncated"] is True
+    assert span.attributes["gen_ai.tool.call.result.original_size_bytes"] == original
+    assert len(payload.encode("utf-8")) <= BOUND
+    envelope = json.loads(payload)
+    assert envelope["_truncated"] is True
+    assert envelope["_original_size_bytes"] == original
+    # Cut as little as possible: the preview fills the bound, it is not a
+    # 4 KB sample of a value that was barely over it.
+    assert len(payload.encode("utf-8")) > BOUND * 0.99
+    assert json.dumps(result).startswith(envelope["_preview"])
+
+
+def _task_span(raw: str):
+    provider, providers, ctx, exporter = _pipeline()
+    agent = Agent(role="Analyst", goal="g", backstory="b")
+    task = Task(description="d", expected_output="e", agent=agent)
+    agent.crew = Crew(agents=[agent], tasks=[task])
+    started = TaskStartedEvent(context=None, task=task)
+    handlers.handle_task_started(providers, ctx, task, started)
+    handlers.handle_task_completed(
+        providers,
+        ctx,
+        task,
+        TaskCompletedEvent(
+            output=TaskOutput(description="d", raw=raw, agent="Analyst"),
+            task=task,
+            started_event_id=started.event_id,
+        ),
+    )
+    span = _only_span(exporter, "execute task")
+    provider.shutdown()
+    return span
+
+
+def test_a_task_output_arrives_whole_under_both_keys() -> None:
+    raw = _text(300_000, label="summary")
+
+    span = _task_span(raw)
+
+    assert span.attributes["crewai.task.output"] == raw
+    assert "crewai.task.output.truncated" not in span.attributes
+    messages = json.loads(span.attributes["gen_ai.output.messages"])
+    assert messages[0]["parts"][0]["content"] == raw
+    assert "gen_ai.output.messages.truncated" not in span.attributes
+
+
+def test_a_plain_attribute_over_the_bound_marks_itself_too() -> None:
+    """``crewai.task.output`` used to have no bound at all: one huge output made
+    the whole span too large for Wharf, which then never stored it."""
+    raw = _text(BOUND + 50_000, label="summary")
+
+    span = _task_span(raw)
+
+    assert span.attributes["crewai.task.output.truncated"] is True
+    assert span.attributes["crewai.task.output.original_size_bytes"] == len(
+        raw.encode("utf-8")
+    )
+    assert len(span.attributes["crewai.task.output"].encode("utf-8")) <= BOUND
+    assert span.attributes["gen_ai.output.messages.truncated"] is True
+
+
+def test_llm_messages_and_response_arrive_whole() -> None:
+    tool_result = _text(300_000)
+    messages = [
+        {"role": "system", "content": "You are an analyst."},
+        {"role": "user", "content": "Summarise the cycle."},
+        {"role": "tool", "content": tool_result},
+    ]
+    answer = _text(100_000, label="answer")
+
+    attrs = semantic_conventions.gen_ai(
+        input_messages=messages, output_messages=answer
+    )
+
+    shaped = json.loads(attrs["gen_ai.input.messages"])
+    assert shaped[-1]["parts"][0]["content"] == tool_result
+    assert json.loads(attrs["gen_ai.output.messages"])[0]["parts"][0][
+        "content"
+    ] == answer
+    assert not any(key.endswith(".truncated") for key in attrs)
+
+
+def test_a_conversation_over_the_bound_keeps_its_ends_and_names_the_cut() -> None:
+    """The repeated conversation on an LLM call is where a large tool result
+    would otherwise be copied into every later call: past the bound its middle
+    is replaced by a placeholder; the tool span keeps the result whole."""
+    messages = [{"role": "system", "content": "You are an analyst."}]
+    messages += [{"role": "tool", "content": _text(200_000, f"r{i}")} for i in range(3)]
+    messages.append({"role": "user", "content": "Now write the summary."})
+
+    attrs = semantic_conventions.gen_ai(input_messages=messages)
+
+    payload = attrs["gen_ai.input.messages"]
+    assert attrs["gen_ai.input.messages.truncated"] is True
+    assert len(payload.encode("utf-8")) <= BOUND
+    shaped = json.loads(payload)
+    assert shaped[0]["parts"][0]["content"] == "You are an analyst."
+    assert shaped[-1]["parts"][0]["content"] == "Now write the summary."
+    assert "[truncated 3 messages" in shaped[1]["parts"][0]["content"]
+
+
+def test_one_cut_message_keeps_all_but_the_overshoot() -> None:
+    """A single message over the bound loses what is over, not half of itself."""
+    content = _text(BOUND + 20_000)
+
+    attrs = semantic_conventions.gen_ai(output_messages=content)
+
+    payload = attrs["gen_ai.output.messages"]
+    assert attrs["gen_ai.output.messages.truncated"] is True
+    assert len(payload.encode("utf-8")) <= BOUND
+    kept = json.loads(payload)[0]["parts"][0]["content"]
+    assert "...[truncated " in kept
+    assert len(payload.encode("utf-8")) > BOUND * 0.99
+
+
+def test_an_sdk_length_limit_lowers_the_bound_so_the_sdk_never_cuts(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The OpenTelemetry SDK cuts a value over its limit with no marker. A user
+    who set that limit gets it, but cut by crewAI, which says so."""
+    limit = 20_000
+    monkeypatch.setenv("OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT", str(limit))
+    assert gen_ai_shapes.max_attr_bytes() == limit
+
+    span = _tool_span(_text(50_000))
+
+    payload = span.attributes["gen_ai.tool.call.result"]
+    assert span.attributes["gen_ai.tool.call.result.truncated"] is True
+    assert len(payload) <= limit
+    # Still valid JSON: crewAI's envelope, not a string the SDK chopped.
+    assert json.loads(payload)["_truncated"] is True
+
+
+def test_the_span_limit_wins_over_the_general_one(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setenv("OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT", "10000")
+    monkeypatch.setenv("OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT", "30000")
+    assert gen_ai_shapes.max_attr_bytes() == 30_000
+    monkeypatch.setenv("CREWAI_OTEL_MAX_ATTR_BYTES", "5000")
+    assert gen_ai_shapes.max_attr_bytes() == 5_000
+
+
+def test_a_lower_crewai_setting_is_honoured(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setenv("CREWAI_OTEL_MAX_ATTR_BYTES", "65536")
+    assert gen_ai_shapes.max_attr_bytes() == 65_536
+    monkeypatch.setenv("CREWAI_OTEL_MAX_ATTR_BYTES", "not a number")
+    assert gen_ai_shapes.max_attr_bytes() == BOUND
+
+
+def test_a_higher_crewai_setting_is_clamped_to_the_ceiling_with_one_warning(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """Above the ceiling, several bounded attributes would not fit one Wharf
+    request; the setting is clamped, said once, never silently raised."""
+    gen_ai_shapes._warn_clamped.cache_clear()
+    monkeypatch.setenv("CREWAI_OTEL_MAX_ATTR_BYTES", "1048576")
+
+    with caplog.at_level("WARNING", logger=gen_ai_shapes.__name__):
+        assert gen_ai_shapes.max_attr_bytes() == BOUND
+        assert gen_ai_shapes.max_attr_bytes() == BOUND
+
+    warnings = [r for r in caplog.records if "CREWAI_OTEL_MAX_ATTR_BYTES" in r.message]
+    assert len(warnings) == 1
+    assert "1048576" in warnings[0].message and str(BOUND) in warnings[0].message
+    assert "3,072,000" in warnings[0].message
+
+
+def test_an_sdk_limit_above_the_ceiling_does_not_raise_the_bound(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setenv("OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT", "2000000")
+    assert gen_ai_shapes.max_attr_bytes() == BOUND
+
+
+def test_a_span_with_every_content_attribute_at_the_bound_fits_one_wharf_request() -> (
+    None
+):
+    """Wharf refuses a request over 3,072,000 encoded bytes and the exporter
+    drops a span that alone is over it. An LLM call carries the most content
+    attributes, so seven of them at the bound must still fit."""
+    provider, providers, ctx, exporter = _pipeline()
+    span = providers.get_tracer().start_span("call llm")
+    big = _text(BOUND * 2)
+    handlers._set_span_attributes(
+        span,
+        {
+            **semantic_conventions.gen_ai(
+                input_messages=[{"role": "user", "content": big}],
+                output_messages=big,
+                system_instructions=big,
+                tool_definitions=[{"name": "t", "description": big}],
+            ),
+            "crewai.a": big,
+            "crewai.b": big,
+            "crewai.c": big,
+        },
+    )
+    span.end()
+    finished = exporter.get_finished_spans()
+    provider.shutdown()
+
+    assert sum(1 for key in finished[0].attributes if key.endswith(".truncated")) == 7
+    assert encode_spans(finished).ByteSize() < MAX_EXPORT_BODY_BYTES
+
+
+def test_a_role_bearing_plain_attribute_is_cut_plainly_never_reshaped() -> None:
+    """A task may produce a JSON list whose items carry ``role`` (a roster, a
+    chat log). On ``crewai.task.output`` that is data, not a GenAI
+    conversation: over the bound it keeps its head, nothing is replaced."""
+    roster = json.dumps(
+        [
+            {"role": "engineer", "name": f"person-{i}", "team": f"team-{i % 7}"}
+            for i in range(12_000)
+        ]
+    )
+    assert len(roster.encode("utf-8")) > BOUND
+
+    span = _task_span(roster)
+
+    kept = span.attributes["crewai.task.output"]
+    assert span.attributes["crewai.task.output.truncated"] is True
+    assert span.attributes["crewai.task.output.original_size_bytes"] == len(
+        roster.encode("utf-8")
+    )
+    assert len(kept.encode("utf-8")) == BOUND
+    assert roster.startswith(kept)
+    assert "truncated" not in kept
+
+
+def test_an_sdk_span_limit_of_zero_is_kept_and_cuts_everything_with_the_marker(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The SDK accepts 0 and cuts every string to nothing; crewAI does the
+    same cut first, so it carries the marker."""
+    monkeypatch.setenv("OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT", "0")
+    monkeypatch.setenv("OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT", "20000")
+    assert gen_ai_shapes.max_attr_bytes() == 0
+
+    span = _task_span("a summary")
+
+    assert span.attributes["crewai.task.output"] == ""
+    assert span.attributes["crewai.task.output.truncated"] is True
+    assert span.attributes["crewai.task.output.original_size_bytes"] == len(
+        "a summary"
+    )
+
+
+def test_an_explicitly_empty_span_limit_is_unlimited_not_the_general_limit(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """In the SDK an empty span setting means unlimited and wins over the
+    general one; the bound is then crewAI's own."""
+    monkeypatch.setenv("OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT", "")
+    monkeypatch.setenv("OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT", "20000")
+    assert gen_ai_shapes.max_attr_bytes() == BOUND
+
+    result = _text(50_000)
+    span = _tool_span(result)
+
+    assert json.loads(span.attributes["gen_ai.tool.call.result"]) == result
+    assert "gen_ai.tool.call.result.truncated" not in span.attributes

lib/crewai/tests/telemetry/test_span_task_agent_tool_attributes.py

@@ -39,7 +39,7 @@ import pytest
 
 
 # Long and non-repeating, so a truncated or elided copy cannot compare equal.
-# Under the default 32 KiB attribute cap, so it must arrive whole.
+# Under the default attribute bound, so it must arrive whole.
 LONG_TEXT = "".join(
     f"paragraph {i}: the quick brown fox jumps over the lazy dog\n" for i in range(400)
 )

lib/crewai/tests/telemetry/test_trace_lifecycle.py

@@ -19,6 +19,9 @@ from crewai.telemetry.tracing.grants import (
     GrantSpanExporter,
     TraceGrantClient,
     TraceGrantError,
+    resolve_tracing_credential,
+    tracing_credential,
+    tracing_credential_source,
 )
 from opentelemetry import trace
 from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
@@ -322,15 +325,115 @@ def test_explicit_disable_prevents_first_time_collection(monkeypatch, disabled):
 
 
 @pytest.mark.parametrize("async_run", [False, True])
-def test_grant_failure_restores_execution_context(monkeypatch, async_run):
+@pytest.mark.parametrize(
+    ("status", "says"),
+    [
+        (401, "refused the CREWAI_USER_PAT token (HTTP 401)"),
+        (403, "Replace it with a valid personal access token"),
+        (503, "unaffected"),
+    ],
+)
+def test_a_refused_grant_runs_untraced_and_says_why(
+    monkeypatch, caplog, async_run, status, says
+):
+    """A trace is a record of the run, not a condition of it: an expired login
+    must not fail the crew. The run completes, nothing is uploaded, the context
+    is restored, and the warning names the fix."""
     monkeypatch.setenv("CREWAI_USER_PAT", "invalid")
+    create = Mock(side_effect=TraceGrantError("AMP rejected credential", status))
+    monkeypatch.setattr(TraceGrantClient, "create", create)
+    exporter = Mock(return_value=InMemorySpanExporter())
+    monkeypatch.setattr(GrantSpanExporter, "_exporter", staticmethod(exporter))
+
+    with caplog.at_level("WARNING", logger="crewai.execution"):
+        flow = ExampleFlow(tracing=True)
+        result = asyncio.run(flow.kickoff_async()) if async_run else flow.kickoff()
+
+    assert result == "hello world"
+    create.assert_called_once()
+    exporter.assert_not_called()
+    assert get_trace_session() is None and get_execution_uuid() is None
+    assert any(says in record.getMessage() for record in caplog.records)
+
+
+@pytest.mark.parametrize(
+    ("source", "says", "never"),
+    [
+        ("pat", "refused the CREWAI_USER_PAT token", "crewai login"),
+        ("integration", "refused the platform integration token", "crewai login"),
+        ("login", "refused the saved login (HTTP 401). Run `crewai login` again", "CREWAI_USER_PAT"),
+    ],
+)
+def test_the_warning_names_the_credential_that_was_refused(
+    monkeypatch, caplog, source, says, never
+):
+    """The fix named is for the credential that was sent: refreshing a login does
+    nothing for a rejected CREWAI_USER_PAT or an integration token."""
+    grants = "crewai.telemetry.tracing.grants"
+    monkeypatch.setattr(
+        f"{grants}.resolve_tracing_credential", lambda: (source, "rejected")
+    )
     monkeypatch.setattr(
         TraceGrantClient,
         "create",
         Mock(side_effect=TraceGrantError("AMP rejected credential", 401)),
     )
-    with pytest.raises(TraceGrantError) as error:
-        flow = ExampleFlow(tracing=True)
-        asyncio.run(flow.kickoff_async()) if async_run else flow.kickoff()
-    assert error.value.status_code == 401
+
+    with caplog.at_level("WARNING", logger="crewai.execution"):
+        assert ExampleFlow(tracing=True).kickoff() == "hello world"
+
+    warning = next(r.getMessage() for r in caplog.records if "not traced" in r.getMessage())
+    assert says in warning and never not in warning
+
+
+def test_the_credential_source_follows_the_credential_order(monkeypatch):
+    grants = "crewai.telemetry.tracing.grants"
+    monkeypatch.setenv("CREWAI_USER_PAT", "pat")
+    assert tracing_credential_source() == "pat"
+    monkeypatch.delenv("CREWAI_USER_PAT")
+    monkeypatch.setattr(f"{grants}.get_platform_integration_token", lambda: "integration")
+    assert tracing_credential_source() == "integration"
+    monkeypatch.setattr(f"{grants}.get_platform_integration_token", lambda: None)
+    monkeypatch.setattr(f"{grants}.get_auth_token", lambda: "login")
+    assert tracing_credential_source() == "login"
+    assert tracing_credential() == "login"
+    assert resolve_tracing_credential() == ("login", "login")
+
+
+def test_the_warning_names_the_credential_that_was_sent_not_the_one_left_after(
+    monkeypatch, caplog
+):
+    """The credential and its source are read once: a PAT that disappears while
+    the grant request is in flight is still the one the warning names, never
+    the integration token the environment falls back to afterwards."""
+    grants = "crewai.telemetry.tracing.grants"
+    monkeypatch.setenv("CREWAI_USER_PAT", "rejected-pat")
+    monkeypatch.setattr(f"{grants}.get_platform_integration_token", lambda: "integration")
+    sent = []
+
+    def refuse(client, execution_uuid):
+        sent.append(client._api.api_key)
+        monkeypatch.delenv("CREWAI_USER_PAT")
+        raise TraceGrantError("AMP rejected credential", 401)
+
+    monkeypatch.setattr(TraceGrantClient, "create", refuse)
+    with caplog.at_level("WARNING", logger="crewai.execution"):
+        assert ExampleFlow(tracing=True).kickoff() == "hello world"
+
+    warning = next(r.getMessage() for r in caplog.records if "not traced" in r.getMessage())
+    assert sent == ["rejected-pat"]
+    assert "CREWAI_USER_PAT" in warning and "integration" not in warning
+
+
+def test_a_blank_credential_runs_untraced_too(monkeypatch, caplog):
+    """The grant client refuses a blank credential in its constructor, with the
+    same error a refused grant raises — so it must not fail the run either."""
+    monkeypatch.setattr(
+        "crewai.telemetry.tracing.grants.resolve_tracing_credential",
+        lambda: ("pat", "   "),
+    )
+    with caplog.at_level("WARNING", logger="crewai.execution"):
+        assert ExampleFlow(tracing=True).kickoff() == "hello world"
     assert get_trace_session() is None and get_execution_uuid() is None
+    assert any("not traced" in record.getMessage() for record in caplog.records)
+

lib/crewai/tests/telemetry/test_tracer_isolation.py

@@ -118,6 +118,32 @@ def test_a_feature_span_keeps_only_what_the_feature_may_send(telemetry_with_expo
     assert deploy_span.attributes["feature"] == "cli_usage:deploy"
 
 
+def test_the_models_span_carries_the_models_compared_and_nothing_else(
+    telemetry_with_exporter,
+):
+    telemetry, exporter = telemetry_with_exporter
+    telemetry.set_tracer()
+
+    telemetry.feature_usage_span(
+        "cli_usage:eval_models",
+        {
+            "authenticated": "true",
+            "models": "openai/gpt-4o-mini,anthropic/claude-haiku-4-5",
+            "models_count": "2",
+            "evaluation_id": "ev-1",
+            "organization_id": "org-1",
+        },
+    )
+
+    (span,) = exporter.get_finished_spans()
+    assert span.attributes["feature"] == "cli_usage:eval_models"
+    assert span.attributes["authenticated"] == "true"
+    assert span.attributes["models"] == "openai/gpt-4o-mini,anthropic/claude-haiku-4-5"
+    assert span.attributes["models_count"] == "2"
+    assert "evaluation_id" not in span.attributes
+    assert "organization_id" not in span.attributes
+
+
 def test_our_spans_are_unaffected_by_an_application_provider(telemetry_with_exporter):
     """An app that installs its own provider must not divert our telemetry.
 

lib/crewai/tests/test_llm.py

@@ -11,7 +11,8 @@ from crewai.events.event_types import (
     ToolUsageFinishedEvent,
     ToolUsageStartedEvent,
 )
-from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO, DEFAULT_CONTEXT_WINDOW_SIZE, LLM
+from crewai.llm import DEFAULT_CONTEXT_WINDOW_SIZE, LLM
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.llms.providers.anthropic.completion import AnthropicCompletion
 from crewai.utilities.agent_utils import is_context_length_exceeded
 from crewai.utilities.exceptions.context_window_exceeding_exception import (
@@ -343,9 +344,9 @@ def test_context_window_validation():
             {"test-model": 500},
             clear=True,
         ):
-            llm = LLM(model="test-model")
+            llm = LLM(model="test-model", is_litellm=True)
             llm.get_context_window_size()
-    assert "must be between 1024 and 2097152" in str(excinfo.value)
+    assert "must be between 1024 and 10000000" in str(excinfo.value)
 
 
 @pytest.mark.parametrize(
@@ -361,7 +362,43 @@ def test_gpt56_family_uses_official_context_window(model: str) -> None:
 def test_gpt56_does_not_override_gpt54_mini_window() -> None:
     """A more specific older prefix must keep its own window."""
     llm = LLM(model="gpt-5.4-mini", is_litellm=True)
-    assert llm.get_context_window_size() == int(200000 * CONTEXT_WINDOW_USAGE_RATIO)
+    assert llm.get_context_window_size() == int(400000 * CONTEXT_WINDOW_USAGE_RATIO)
+
+
+def test_qualified_litellm_gpt4_keeps_the_provider_context_window() -> None:
+    openai_llm = LLM(model="openai/gpt-4", is_litellm=True)
+    azure_llm = LLM(model="azure/gpt-4", is_litellm=True)
+
+    assert openai_llm.get_context_window_size() == int(8192 * CONTEXT_WINDOW_USAGE_RATIO)
+    assert azure_llm.get_context_window_size() == int(
+        128_000 * CONTEXT_WINDOW_USAGE_RATIO
+    )
+
+
+def test_litellm_gpt5_uses_the_shared_openai_context_window() -> None:
+    llm = LLM(model="gpt-5", is_litellm=True)
+
+    assert llm.get_context_window_size() == int(
+        400_000 * CONTEXT_WINDOW_USAGE_RATIO
+    )
+
+
+@pytest.mark.parametrize(
+    ("native_model", "litellm_model"),
+    [
+        ("openai/gpt-5", "gpt-5"),
+        ("openai/o1-pro", "o1-pro"),
+        ("anthropic/claude-sonnet-4-6", "claude-sonnet-4-6"),
+        ("google/gemini-2.5-flash", "gemini-2.5-flash"),
+    ],
+)
+def test_native_and_litellm_context_windows_match(
+    native_model: str, litellm_model: str
+) -> None:
+    native_llm = LLM(model=native_model)
+    litellm_llm = LLM(model=litellm_model, is_litellm=True)
+
+    assert native_llm.get_context_window_size() == litellm_llm.get_context_window_size()
 
 
 @pytest.mark.parametrize(

lib/crewai/tests/test_llm_overlay.py

@@ -1,10 +1,13 @@
-"""`llm_overlay` swaps an agent's model by role, for the calling context only.
+"""`llm_overlay` swaps an agent's model by role, and any LLM's by model, for
+the calling context only.
 
-The overlay is read in exactly three places: the validators where `Agent` and
-`LiteAgent` resolve their `llm`, and `Agent.interpolate_inputs`, which looks the
+The overlay is read in exactly four places: the validators where `Agent` and
+`LiteAgent` resolve their `llm`, `Agent.interpolate_inputs`, which looks the
 interpolated role up again because a templated role only becomes a key once a
-kickoff fills its placeholders in. So these tests build agents, interpolate
-them, and look at the model they end up with. No LLM is ever called.
+kickoff fills its placeholders in, and `LLM.__new__`, where a `model:` key maps
+an LLM built from a model string. So these tests build agents and LLMs,
+interpolate them, and look at the model they end up with. No LLM is ever called
+over the network.
 
 `create_llm("openai/gpt-4o")` returns the native OpenAI provider, which strips
 the `openai/` prefix, so the resolved model reads `"gpt-4o"`.
@@ -19,8 +22,14 @@ from typing import Any
 from crewai import Agent, Crew, Task
 from crewai.lite_agent import LiteAgent
 from crewai.llm import LLM
-from crewai.llm_overlay import active, llm_overlay, overlay_model_for
+from crewai.llm_overlay import (
+    MODEL_KEY_PREFIX,
+    active,
+    llm_overlay,
+    overlay_model_for,
+)
 from crewai.llms.base_llm import BaseLLM
+from crewai.utilities.llm_utils import create_llm
 import pytest
 
 
@@ -567,3 +576,315 @@ def test_re_validation_keeps_the_llm_a_kickoff_time_swap_set() -> None:
         RuntimeState(root=[agent])
 
     assert agent.llm is swapped
+
+
+# ── model keys: `model:<provider/model>` and `model:*` ───────────────────────
+
+
+def test_the_model_key_prefix_is_a_public_constant() -> None:
+    """Another package feature-detects the model-for-model form on it."""
+    assert MODEL_KEY_PREFIX == "model:"
+
+
+def test_a_bare_llm_call_in_a_flow_step_runs_on_the_mapped_model(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The case roles cannot reach: a flow step's own `LLM(...).call()`."""
+    from crewai.flow.flow import Flow, start
+
+    openai_class = type(LLM(model="openai/gpt-4o-mini"))
+    monkeypatch.setattr(
+        openai_class,
+        "call",
+        lambda self, messages, *args, **kwargs: f"answered by {self.model}",
+    )
+
+    class Poem(Flow):  # type: ignore[type-arg]
+        @start()
+        def write(self) -> str:
+            return LLM(model="openai/gpt-5.6-sol").call("a poem")
+
+    with llm_overlay({"model:openai/gpt-5.6-sol": "openai/gpt-4o-mini"}):
+        assert Poem().kickoff() == "answered by gpt-4o-mini"
+
+    assert Poem().kickoff() == "answered by gpt-5.6-sol"
+
+
+def test_a_model_key_matches_with_and_without_the_provider_prefix() -> None:
+    """Native providers strip `openai/`; a key written either way matches both."""
+    with llm_overlay({"model:openai/gpt-4o": "openai/gpt-4.1"}):
+        assert LLM(model="gpt-4o").model == "gpt-4.1"
+        assert LLM(model="openai/gpt-4o").model == "gpt-4.1"
+        assert LLM(model="openai/gpt-4o-mini").model == "gpt-4o-mini"
+    with llm_overlay({"model:gpt-4o": "openai/gpt-4.1"}):
+        assert LLM(model="openai/gpt-4o").model == "gpt-4.1"
+
+
+def test_an_exact_model_key_wins_over_a_stripped_one_and_over_the_wildcard() -> (
+    None
+):
+    with llm_overlay(
+        {
+            "model:gpt-4o": "openai/gpt-4.1-nano",
+            "model:openai/gpt-4o": "openai/gpt-4.1",
+            "model:*": "openai/gpt-4o-mini",
+        }
+    ):
+        assert LLM(model="openai/gpt-4o").model == "gpt-4.1"
+        assert LLM(model="gpt-4o").model == "gpt-4.1-nano"
+        assert LLM(model="openai/o3-mini").model == "gpt-4o-mini"
+
+
+def test_the_wildcard_maps_every_llm_built_from_a_model_string() -> None:
+    with llm_overlay({"model:*": "openai/gpt-4o-mini"}):
+        built = [
+            LLM(model="openai/gpt-4o"),
+            create_llm("anthropic/claude-haiku-4-5"),
+            LLM(model="gpt-5.6-sol"),
+        ]
+
+    assert [(type(b).__name__, b.model) for b in built] == [
+        ("OpenAICompletion", "gpt-4o-mini")
+    ] * 3
+    assert LLM(model="openai/gpt-4o").model == "gpt-4o"
+
+
+def test_a_mapped_model_is_not_mapped_again() -> None:
+    """A chain of keys is one step: the model a key maps to is built as it is."""
+    with llm_overlay(
+        {"model:openai/gpt-4o": "openai/gpt-4.1", "model:openai/gpt-4.1": "openai/o3"}
+    ):
+        assert LLM(model="openai/gpt-4o").model == "gpt-4.1"
+
+
+def test_the_caller_settings_follow_the_mapped_model_by_the_declared_llm_rule() -> (
+    None
+):
+    """Generation settings go to any provider; a key and an endpoint only to
+    the provider they were issued for."""
+    with llm_overlay({"model:*": "openai/gpt-4o"}):
+        same = LLM(model="openai/gpt-4o-mini", **CONFIGURATION)
+    with llm_overlay({"model:*": "anthropic/claude-haiku-4-5"}):
+        other = LLM(model="openai/gpt-4o-mini", **CONFIGURATION)
+
+    assert type(same).__name__ == "OpenAICompletion" and same.model == "gpt-4o"
+    assert _configuration_of(same) == CONFIGURATION
+    assert type(other).__name__ == "AnthropicCompletion"
+    assert other.model == "claude-haiku-4-5"
+    assert other.timeout == 42 and other.temperature == 0.1
+    assert other.max_tokens == 77
+    assert other.api_key != "k" and other.base_url is None
+
+
+def test_a_model_mapped_onto_litellm_is_not_initialized_again() -> None:
+    """`LLM.__new__` returning an `LLM` makes Python call `__init__` with the
+    caller's arguments; the mapped instance must keep the mapped model."""
+    pytest.importorskip("litellm")
+    with llm_overlay({"model:*": "groq/llama-3.1-8b-instant"}):
+        built = LLM(model="openai/gpt-4o-mini", temperature=0.3, api_key="k")
+
+    assert type(built).__name__ == "LLM" and built.is_litellm
+    assert built.model == "groq/llama-3.1-8b-instant"
+    assert built.temperature == 0.3 and built.api_key != "k"
+
+
+def test_an_agent_declared_with_a_model_string_runs_on_the_mapped_model() -> None:
+    with llm_overlay({"model:openai/gpt-4o-mini": "openai/gpt-4o"}):
+        agent = _agent("Writer")
+
+    assert agent.llm.model == "gpt-4o"
+
+
+def test_an_agent_whose_llm_was_built_outside_the_block_is_mapped_by_its_model() -> (
+    None
+):
+    """The declared instance is looked up by its model, with the configuration
+    carried like a role's swap."""
+    declared = _configured_llm()
+    with llm_overlay({"model:openai/gpt-4o-mini": "openai/gpt-4o"}):
+        agent = Agent(role="Writer", goal="g", backstory="b", llm=declared)
+        lite = LiteAgent(role="Writer", goal="g", backstory="b", llm=declared)
+
+    for built in (agent.llm, lite.llm):
+        assert built is not declared and built.model == "gpt-4o"
+        assert _configuration_of(built) == CONFIGURATION
+
+
+def test_a_role_key_wins_over_model_keys_for_that_agent() -> None:
+    with llm_overlay(
+        {
+            "Researcher": "openai/gpt-4o",
+            "model:*": "openai/gpt-4.1-nano",
+        }
+    ):
+        researcher = _agent("Researcher")
+        writer = _agent("Writer")
+        outside = Agent(
+            role="Researcher", goal="g", backstory="b", llm=_configured_llm()
+        )
+
+    assert researcher.llm.model == "gpt-4o"
+    assert writer.llm.model == "gpt-4.1-nano"
+    assert outside.llm.model == "gpt-4o"
+
+
+def test_a_role_that_looks_like_a_model_key_is_never_one() -> None:
+    with llm_overlay({"model:*": "openai/gpt-4o"}):
+        assert overlay_model_for("model:*") is None
+
+
+def test_a_model_key_that_names_no_model_is_refused() -> None:
+    with pytest.raises(ValueError, match="names no model"):
+        with llm_overlay({"model: ": "openai/gpt-4o"}):
+            pass
+
+
+def test_a_subclass_of_llm_keeps_its_model() -> None:
+    """Only `LLM` itself routes; a subclass is the caller's own choice of class."""
+    pytest.importorskip("litellm")
+
+    class Mine(LLM):
+        pass
+
+    with llm_overlay({"model:*": "openai/gpt-4o"}):
+        mine = Mine(model="groq/llama-3.1-8b-instant")
+
+    assert type(mine) is Mine and mine.model == "groq/llama-3.1-8b-instant"
+
+
+def test_a_declared_model_whose_sdk_is_missing_still_maps(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Nothing is built on the declared model, so its SDK need not be here; its
+    key cannot be matched to the new provider, so it stays behind."""
+    native = LLM._get_native_provider.__func__  # type: ignore[attr-defined]
+
+    def without_anthropic(cls: type[LLM], provider: str) -> Any:
+        if provider in ("anthropic", "claude"):
+            raise ImportError("Anthropic native provider not available")
+        return native(cls, provider)
+
+    monkeypatch.setattr(LLM, "_get_native_provider", classmethod(without_anthropic))
+    with llm_overlay({"model:*": "openai/gpt-4o-mini"}):
+        built = LLM(model="anthropic/claude-haiku-4-5", api_key="k", temperature=0.2)
+
+    assert type(built).__name__ == "OpenAICompletion" and built.model == "gpt-4o-mini"
+    assert built.temperature == 0.2 and built.api_key != "k"
+
+
+def test_a_litellm_routed_llm_keeps_its_key_on_the_same_provider_only() -> None:
+    """Same provider is a question about the provider, not the class: an
+    `LLM(...)` the caller routed through LiteLLM keeps its key and endpoint
+    when mapped to another model of that provider, and another provider's
+    model never gets them."""
+    pytest.importorskip("litellm")
+    declared = {"api_key": "k", "base_url": "http://localhost:9999/v1"}
+    with llm_overlay({"model:*": "openai/gpt-4o-mini"}):
+        same = LLM(model="openai/gpt-4o", is_litellm=True, **declared)
+    with llm_overlay({"model:*": "anthropic/claude-haiku-4-5"}):
+        other = LLM(model="openai/gpt-4o", is_litellm=True, **declared)
+
+    assert "gpt-4o-mini" in same.model
+    assert same.api_key == "k" and same.base_url == "http://localhost:9999/v1"
+    assert "claude-haiku-4-5" in other.model
+    assert other.api_key != "k" and other.base_url != "http://localhost:9999/v1"
+
+
+@pytest.mark.parametrize(
+    ("built", "key"),
+    [
+        ("openrouter/openai/gpt-4o", "model:openai/gpt-4o"),
+        ("openrouter/openai/gpt-4o", "model:gpt-4o"),
+    ],
+)
+def test_an_aggregators_route_is_never_the_native_model_it_names(
+    built: str, key: str
+) -> None:
+    """`openrouter/openai/gpt-4o` is OpenRouter's model, not OpenAI's: a key
+    written for the native model leaves it alone."""
+    with llm_overlay({key: "openai/gpt-4.1"}):
+        llm = LLM(model=built)
+
+    assert type(llm).__name__ == "OpenAICompatibleCompletion"
+    assert llm.model == "openai/gpt-4o"
+
+
+def test_an_aggregators_route_is_matched_by_its_own_full_name() -> None:
+    declared = LLM(model="openrouter/openai/gpt-4o")
+    with llm_overlay({"model:openrouter/openai/gpt-4o": "openai/gpt-4.1"}):
+        built = LLM(model="openrouter/openai/gpt-4o")
+        agent = Agent(role="Writer", goal="g", backstory="b", llm=declared)
+    with llm_overlay({"model:openai/gpt-4o": "openai/gpt-4.1"}):
+        untouched = Agent(role="Writer", goal="g", backstory="b", llm=declared)
+
+    assert built.model == "gpt-4.1" and agent.llm.model == "gpt-4.1"
+    assert untouched.llm is declared
+
+
+def test_a_native_provider_prefix_is_still_its_own() -> None:
+    with llm_overlay({"model:llama3": "ollama/qwen3"}):
+        assert LLM(model="ollama/llama3").model == "qwen3"
+
+
+def test_a_role_key_is_built_from_the_declaration_a_model_key_mapped() -> None:
+    """Role wins, with the caller's declared settings: the llm a model key
+    swapped on the way in is not what the role's model is built like."""
+    overlay = {"Researcher": "openai/gpt-4o", "model:*": "anthropic/claude-haiku-4-5"}
+    with llm_overlay(overlay):
+        declared = _configured_llm()  # mapped to Anthropic, without the key
+        researcher = Agent(role="Researcher", goal="g", backstory="b", llm=declared)
+        by_string = Agent(
+            role="Researcher", goal="g", backstory="b", llm="openai/gpt-4o-mini"
+        )
+        writer = Agent(role="Writer", goal="g", backstory="b", llm=declared)
+
+    assert type(declared).__name__ == "AnthropicCompletion"
+    assert type(researcher.llm).__name__ == "OpenAICompletion"
+    assert researcher.llm.model == "gpt-4o"
+    assert _configuration_of(researcher.llm) == CONFIGURATION
+    assert by_string.llm.model == "gpt-4o"
+    assert writer.llm is declared
+
+
+def test_a_providers_other_name_is_the_same_model_on_both_paths(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """`google/x` routes as `gemini/x`, so a key written either way matches an
+    llm built in the block and one built before it, whichever name it used."""
+    pytest.importorskip("google.genai")
+    monkeypatch.setenv("GEMINI_API_KEY", "g")
+    outside = LLM(model="google/gemini-2.5-pro")  # records provider "gemini"
+    for key in ("model:google/gemini-2.5-pro", "model:gemini/gemini-2.5-pro"):
+        with llm_overlay({key: "openai/gpt-4o-mini"}):
+            built = [LLM(model="google/gemini-2.5-pro"), LLM(model="gemini/gemini-2.5-pro")]
+            agent = Agent(role="Writer", goal="g", backstory="b", llm=outside)
+
+        assert [b.model for b in built] == ["gpt-4o-mini", "gpt-4o-mini"], key
+        assert agent.llm.model == "gpt-4o-mini", key
+
+
+def test_an_aggregator_route_under_an_alias_still_is_not_the_native_model() -> None:
+    with llm_overlay({"model:google/gemini-2.5-pro": "openai/gpt-4o-mini"}):
+        llm = LLM(model="openrouter/google/gemini-2.5-pro")
+
+    assert llm.model == "google/gemini-2.5-pro"
+
+
+def test_a_caller_who_chose_litellm_keeps_it_when_the_declared_sdk_is_missing(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    pytest.importorskip("litellm")
+    native = LLM._get_native_provider.__func__  # type: ignore[attr-defined]
+
+    def without_anthropic(cls: type[LLM], provider: str) -> Any:
+        if provider in ("anthropic", "claude"):
+            raise ImportError("Anthropic native provider not available")
+        return native(cls, provider)
+
+    monkeypatch.setattr(LLM, "_get_native_provider", classmethod(without_anthropic))
+    with llm_overlay({"model:*": "openai/gpt-4o-mini"}):
+        chosen = LLM(model="anthropic/claude-haiku-4-5", is_litellm=True)
+        default = LLM(model="anthropic/claude-haiku-4-5")
+
+    assert type(chosen).__name__ == "LLM" and chosen.is_litellm
+    assert type(default).__name__ == "OpenAICompletion"

lib/crewai/tests/tracing/test_tracing.py

@@ -86,8 +86,8 @@ class TestTraceListenerSetup:
         transport = SimpleNamespace(grants=[], exporter=InMemorySpanExporter())
         monkeypatch.setenv("OTEL_SDK_DISABLED", "false")
         monkeypatch.setattr(
-            "crewai.telemetry.tracing.grants.tracing_credential",
-            lambda: "synthetic-login",
+            "crewai.telemetry.tracing.grants.resolve_tracing_credential",
+            lambda: ("login", "synthetic-login"),
         )
 
         def grant(client, execution_uuid):
@@ -441,7 +441,7 @@ class TestTraceListenerSetup:
     def test_trace_listener_ephemeral_batch(self, trace_transport, monkeypatch):
         """Unauthenticated kickoff uploads buffered spans only after consent."""
         monkeypatch.setattr(
-            "crewai.telemetry.tracing.grants.tracing_credential", lambda: None
+            "crewai.telemetry.tracing.grants.resolve_tracing_credential", lambda: None
         )
         monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
 

lib/crewai/tests/utilities/test_agent_utils.py

@@ -18,18 +18,11 @@ from crewai.hooks.tool_hooks import (
 )
 from crewai.agents.parser import AgentFinish
 from crewai.tools.base_tool import BaseTool
-from crewai.llm import CONTEXT_WINDOW_USAGE_RATIO
+from crewai.llms.context_window import CONTEXT_WINDOW_USAGE_RATIO
 from crewai.utilities.agent_utils import (
-    _asummarize_chunks,
-    _estimate_token_count,
-    _expand_oversized_message,
-    _extract_summary_tags,
-    _format_messages_for_summary,
     message_content_text,
-    _normalize_messages_for_chunking,
-    _split_messages_into_chunks,
-    _split_text_by_token_limit,
     format_message_for_llm,
+    LLMContextLengthExceededError,
     convert_tools_to_openai_schema,
     handle_max_iterations_exceeded,
     execute_single_native_tool_call,
@@ -38,13 +31,20 @@ from crewai.utilities.agent_utils import (
     NativeToolCallResult,
     parse_tool_call_args,
     summarize_messages,
+    SummarizeMessages,
 )
 from crewai.utilities.i18n import I18N_DEFAULT
 
+_summarizer = SummarizeMessages()
+_approx_tokens = _summarizer._approx_tokens
+_conversation_text = _summarizer._conversation_text
+_messages_ready_to_chunk = _summarizer._messages_ready_to_chunk
+_chunk_messages = _summarizer._chunk_messages
+
 
 def _estimate_summarization_request_tokens(chunk: list[dict[str, Any]]) -> int:
     """Estimate tokens for the full summarization LLM request for one chunk."""
-    conversation_text = _format_messages_for_summary(chunk)
+    conversation_text = _conversation_text(chunk)
     summarization_messages = [
         format_message_for_llm(
             I18N_DEFAULT.slice("summarizer_system_message"), role="system"
@@ -56,7 +56,7 @@ def _estimate_summarization_request_tokens(chunk: list[dict[str, Any]]) -> int:
         ),
     ]
     return sum(
-        _estimate_token_count(str(message.get("content", "")))
+        _approx_tokens(str(message.get("content", "")))
         for message in summarization_messages
     )
 
@@ -349,7 +349,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>Summarized conversation about image analysis.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>Summarized conversation about image analysis.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -380,7 +380,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>Summarized conversation.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>Summarized conversation.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -406,7 +406,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>A greeting exchange.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>A greeting exchange.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -429,7 +429,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>Summary</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>Summary</summary>")
 
         summarize_messages(
             messages=messages,
@@ -451,7 +451,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>User asked about AI, assistant found resources.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>User asked about AI, assistant found resources.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -475,7 +475,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>Greeting exchange.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>Greeting exchange.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -484,7 +484,7 @@ class TestSummarizeMessages:
 
         )
 
-        call_args = mock_llm.call.call_args[0][0]
+        call_args = mock_llm.acall.call_args[0][0]
         user_msg_content = call_args[1]["content"]
         assert "[USER]:" in user_msg_content
         assert "[ASSISTANT]:" in user_msg_content
@@ -500,7 +500,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "Here is the summary:\n<summary>The extracted summary content.</summary>\nExtra text."
+        mock_llm.acall = AsyncMock(return_value="Here is the summary:\n<summary>The extracted summary content.</summary>\nExtra text.")
 
         summarize_messages(
             messages=messages,
@@ -524,7 +524,7 @@ class TestSummarizeMessages:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 1000
-        mock_llm.call.return_value = "<summary>User searched for Python info.</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>User searched for Python info.</summary>")
 
         summarize_messages(
             messages=messages,
@@ -533,7 +533,7 @@ class TestSummarizeMessages:
 
         )
 
-        call_args = mock_llm.call.call_args[0][0]
+        call_args = mock_llm.acall.call_args[0][0]
         user_msg_content = call_args[1]["content"]
         assert "[TOOL_RESULT (web_search)]:" in user_msg_content
 
@@ -555,7 +555,7 @@ class TestSummarizeMessages:
         )
 
         # No LLM call should have been made
-        mock_llm.call.assert_not_called()
+        mock_llm.acall.assert_not_called()
         # System messages should remain untouched
         assert len(messages) == 2
         assert messages[0]["content"] == "You are a helpful assistant."
@@ -563,14 +563,14 @@ class TestSummarizeMessages:
 
 
 class TestFormatMessagesForSummary:
-    """Tests for _format_messages_for_summary helper."""
+    """Tests for _conversation_text helper."""
 
     def test_skips_system_messages(self) -> None:
         messages: list[dict[str, Any]] = [
             {"role": "system", "content": "System prompt"},
             {"role": "user", "content": "Hello"},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "System prompt" not in result
         assert "[USER]: Hello" in result
 
@@ -579,7 +579,7 @@ class TestFormatMessagesForSummary:
             {"role": "user", "content": "Question"},
             {"role": "assistant", "content": "Answer"},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "[USER]: Question" in result
         assert "[ASSISTANT]: Answer" in result
 
@@ -587,7 +587,7 @@ class TestFormatMessagesForSummary:
         messages: list[dict[str, Any]] = [
             {"role": "tool", "content": "Result data", "name": "search_tool"},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "[TOOL_RESULT (search_tool)]:" in result
         assert "Result data" in result
 
@@ -597,14 +597,14 @@ class TestFormatMessagesForSummary:
                 {"function": {"name": "calculator", "arguments": "{}"}}
             ]},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "[Called tools: calculator]" in result
 
     def test_handles_none_content_without_tool_calls(self) -> None:
         messages: list[dict[str, Any]] = [
             {"role": "assistant", "content": None},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "[ASSISTANT]:" in result
 
     def test_handles_multimodal_content(self) -> None:
@@ -614,49 +614,59 @@ class TestFormatMessagesForSummary:
                 {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
             ]},
         ]
-        result = _format_messages_for_summary(messages)
+        result = _conversation_text(messages)
         assert "[USER]: Describe this image" in result
 
     def test_empty_messages(self) -> None:
-        result = _format_messages_for_summary([])
+        result = _conversation_text([])
         assert result == ""
 
 
 class TestExtractSummaryTags:
-    """Tests for _extract_summary_tags helper."""
+    """Summary tags are pulled out inside _summarize_all."""
+
+    def _from_reply(self, reply: str) -> str:
+        summarizer = SummarizeMessages()
+        summarizer.llm = MagicMock()
+        summarizer.callbacks = []
+        summarizer.llm.acall = AsyncMock(return_value=reply)
+        results = summarizer._summarize_all(
+            chunks=[[{"role": "user", "content": "x"}]], char_level_index=0
+        )
+        return results[0]
 
     def test_extracts_content_from_tags(self) -> None:
         text = "Preamble\n<summary>The actual summary.</summary>\nPostamble"
-        assert _extract_summary_tags(text) == "The actual summary."
+        assert self._from_reply(text) == "The actual summary."
 
     def test_handles_multiline_content(self) -> None:
         text = "<summary>\nLine 1\nLine 2\nLine 3\n</summary>"
-        result = _extract_summary_tags(text)
+        result = self._from_reply(text)
         assert "Line 1" in result
         assert "Line 2" in result
         assert "Line 3" in result
 
     def test_falls_back_when_no_tags(self) -> None:
         text = "Just a plain summary without tags."
-        assert _extract_summary_tags(text) == text
+        assert self._from_reply(text) == text
 
     def test_handles_empty_string(self) -> None:
-        assert _extract_summary_tags("") == ""
+        assert self._from_reply("") == ""
 
     def test_extracts_first_match(self) -> None:
         text = "<summary>First</summary> text <summary>Second</summary>"
-        assert _extract_summary_tags(text) == "First"
+        assert self._from_reply(text) == "First"
 
 
 class TestSplitMessagesIntoChunks:
-    """Tests for _split_messages_into_chunks helper."""
+    """Tests for _chunk_messages helper."""
 
     def test_single_chunk_when_under_limit(self) -> None:
         messages: list[dict[str, Any]] = [
             {"role": "user", "content": "Hello"},
             {"role": "assistant", "content": "Hi"},
         ]
-        chunks = _split_messages_into_chunks(messages, max_tokens=1000)
+        chunks = _chunk_messages(messages, max_tokens=1000)
         assert len(chunks) == 1
         assert len(chunks[0]) == 2
 
@@ -667,7 +677,7 @@ class TestSplitMessagesIntoChunks:
             {"role": "user", "content": "C" * 100},
         ]
         # max_tokens=30 should cause splits
-        chunks = _split_messages_into_chunks(messages, max_tokens=30)
+        chunks = _chunk_messages(messages, max_tokens=30)
         assert len(chunks) == 3
 
     def test_excludes_system_messages(self) -> None:
@@ -675,21 +685,21 @@ class TestSplitMessagesIntoChunks:
             {"role": "system", "content": "System prompt"},
             {"role": "user", "content": "Hello"},
         ]
-        chunks = _split_messages_into_chunks(messages, max_tokens=1000)
+        chunks = _chunk_messages(messages, max_tokens=1000)
         assert len(chunks) == 1
         for chunk in chunks:
             for msg in chunk:
                 assert msg.get("role") != "system"
 
     def test_empty_messages(self) -> None:
-        chunks = _split_messages_into_chunks([], max_tokens=1000)
+        chunks = _chunk_messages([], max_tokens=1000)
         assert chunks == []
 
     def test_only_system_messages(self) -> None:
         messages: list[dict[str, Any]] = [
             {"role": "system", "content": "System prompt"},
         ]
-        chunks = _split_messages_into_chunks(messages, max_tokens=1000)
+        chunks = _chunk_messages(messages, max_tokens=1000)
         assert chunks == []
 
     def test_handles_none_content(self) -> None:
@@ -697,7 +707,7 @@ class TestSplitMessagesIntoChunks:
             {"role": "assistant", "content": None},
             {"role": "user", "content": "Follow up"},
         ]
-        chunks = _split_messages_into_chunks(messages, max_tokens=1000)
+        chunks = _chunk_messages(messages, max_tokens=1000)
         assert len(chunks) == 1
         assert len(chunks[0]) == 2
 
@@ -706,11 +716,11 @@ class TestSplitMessagesIntoChunks:
             {"role": "tool", "content": "X" * 1200, "name": "web_scraper"},
         ]
         max_tokens = 100
-        chunks = _split_messages_into_chunks(messages, max_tokens=max_tokens)
+        chunks = _chunk_messages(messages, max_tokens=max_tokens)
         assert len(chunks) > 1
         for chunk in chunks:
             chunk_tokens = sum(
-                _estimate_token_count(message_content_text(msg)) for msg in chunk
+                _approx_tokens(message_content_text(msg)) for msg in chunk
             )
             assert chunk_tokens <= max_tokens
 
@@ -721,11 +731,11 @@ class TestSplitMessagesIntoChunks:
             {"role": "assistant", "content": "Done"},
         ]
         max_tokens = 100
-        chunks = _split_messages_into_chunks(messages, max_tokens=max_tokens)
+        chunks = _chunk_messages(messages, max_tokens=max_tokens)
         assert len(chunks) > 1
         for chunk in chunks:
             chunk_tokens = sum(
-                _estimate_token_count(message_content_text(msg)) for msg in chunk
+                _approx_tokens(message_content_text(msg)) for msg in chunk
             )
             assert chunk_tokens <= max_tokens
 
@@ -743,7 +753,7 @@ class TestSplitMessagesIntoChunks:
             {"role": "assistant", "content": "Collected HubSpot results."},
         ]
 
-        chunks = _split_messages_into_chunks(messages, max_tokens=chunk_budget)
+        chunks = _chunk_messages(messages, max_tokens=chunk_budget)
         assert len(chunks) > 1
 
         for chunk in chunks:
@@ -805,84 +815,58 @@ class TestMessageContentText:
         assert "image_url" not in text
 
 
-class TestSplitTextByTokenLimit:
-    """Tests for _split_text_by_token_limit helper."""
+class TestMessagesReadyToChunk:
+    """Tests for _messages_ready_to_chunk."""
 
-    def test_empty_string(self) -> None:
-        assert _split_text_by_token_limit("", max_tokens=100) == []
+    def test_keeps_short_message(self) -> None:
+        messages: list[dict[str, Any]] = [{"role": "user", "content": "hello"}]
+        assert _messages_ready_to_chunk(messages, max_tokens=100) == messages
 
-    def test_under_limit_returns_single_part(self) -> None:
-        assert _split_text_by_token_limit("hello", max_tokens=100) == ["hello"]
-
-    def test_split_preserves_content(self) -> None:
-        text = "a" * 600
-        parts = _split_text_by_token_limit(text, max_tokens=100)
-        assert len(parts) > 1
-        assert "".join(parts) == text
-
-    def test_each_part_estimated_under_limit(self) -> None:
-        text = "b" * 1200
+    def test_splits_long_text_and_keeps_each_part_under_limit(self) -> None:
+        messages: list[dict[str, Any]] = [{"role": "user", "content": "b" * 1200}]
         max_tokens = 100
-        parts = _split_text_by_token_limit(text, max_tokens=max_tokens)
-        assert all(_estimate_token_count(part) <= max_tokens for part in parts)
-
-
-class TestExpandOversizedMessage:
-    """Tests for _expand_oversized_message helper."""
-
-    def test_returns_original_when_under_limit(self) -> None:
-        msg: dict[str, Any] = {"role": "user", "content": "hello"}
-        expanded = _expand_oversized_message(msg, max_tokens=100)
-        assert expanded == [msg]
+        ready = _messages_ready_to_chunk(messages, max_tokens=max_tokens)
+        assert len(ready) > 1
+        assert ready[0]["content"].startswith("[Part 1/")
+        assert all(
+            _approx_tokens(message_content_text(msg)) <= max_tokens for msg in ready
+        )
 
     def test_splits_tool_output_with_metadata(self) -> None:
-        msg: dict[str, Any] = {
-            "role": "tool",
-            "content": "Z" * 1200,
-            "name": "fetch_page",
-            "tool_call_id": "call_123",
-        }
-        expanded = _expand_oversized_message(msg, max_tokens=100)
-        assert len(expanded) > 1
-        assert all(part["role"] == "tool" for part in expanded)
-        assert all(part["name"] == "fetch_page" for part in expanded)
-        assert all(part["tool_call_id"] == "call_123" for part in expanded)
-        assert expanded[0]["content"].startswith("[Part 1/")
+        messages: list[dict[str, Any]] = [
+            {
+                "role": "tool",
+                "content": "Z" * 1200,
+                "name": "fetch_page",
+                "tool_call_id": "call_123",
+            }
+        ]
+        ready = _messages_ready_to_chunk(messages, max_tokens=100)
+        assert len(ready) > 1
+        assert all(part["role"] == "tool" for part in ready)
+        assert all(part["name"] == "fetch_page" for part in ready)
+        assert all(part["tool_call_id"] == "call_123" for part in ready)
 
     def test_preserves_non_content_fields(self) -> None:
         mock_file = MagicMock()
-        msg: dict[str, Any] = {
-            "role": "user",
-            "content": "X" * 1200,
-            "files": {"report.pdf": mock_file},
-        }
-        expanded = _expand_oversized_message(msg, max_tokens=100)
-        assert len(expanded) > 1
-        assert all(part["role"] == "user" for part in expanded)
-        assert all(part["files"] == {"report.pdf": mock_file} for part in expanded)
-
-    def test_each_part_estimated_under_limit(self) -> None:
-        msg: dict[str, Any] = {"role": "user", "content": "Y" * 1200}
-        max_tokens = 100
-        expanded = _expand_oversized_message(msg, max_tokens=max_tokens)
-        assert len(expanded) > 1
-        assert all(
-            _estimate_token_count(message_content_text(part)) <= max_tokens
-            for part in expanded
-        )
-
-
-class TestNormalizeMessagesForChunking:
-    """Tests for _normalize_messages_for_chunking helper."""
+        messages: list[dict[str, Any]] = [
+            {
+                "role": "user",
+                "content": "X" * 1200,
+                "files": {"report.pdf": mock_file},
+            }
+        ]
+        ready = _messages_ready_to_chunk(messages, max_tokens=100)
+        assert len(ready) > 1
+        assert all(part["files"] == {"report.pdf": mock_file} for part in ready)
 
     def test_excludes_system_messages(self) -> None:
         messages: list[dict[str, Any]] = [
             {"role": "system", "content": "System prompt"},
             {"role": "user", "content": "Hello"},
         ]
-        normalized = _normalize_messages_for_chunking(messages, max_tokens=1000)
-        assert len(normalized) == 1
-        assert normalized[0]["role"] == "user"
+        ready = _messages_ready_to_chunk(messages, max_tokens=1000)
+        assert ready == [{"role": "user", "content": "Hello"}]
 
     def test_expands_oversized_and_preserves_small_messages(self) -> None:
         messages: list[dict[str, Any]] = [
@@ -891,32 +875,31 @@ class TestNormalizeMessagesForChunking:
             {"role": "assistant", "content": "Done"},
         ]
         max_tokens = 100
-        normalized = _normalize_messages_for_chunking(messages, max_tokens=max_tokens)
-        assert normalized[0]["content"] == "Short"
-        assert normalized[-1]["content"] == "Done"
-        assert len(normalized) > 3
+        ready = _messages_ready_to_chunk(messages, max_tokens=max_tokens)
+        assert ready[0]["content"] == "Short"
+        assert ready[-1]["content"] == "Done"
+        assert len(ready) > 3
         assert all(
-            _estimate_token_count(message_content_text(msg)) <= max_tokens
-            for msg in normalized
+            _approx_tokens(message_content_text(msg)) <= max_tokens for msg in ready
         )
 
 
 class TestEstimateTokenCount:
-    """Tests for _estimate_token_count helper."""
+    """Tests for _approx_tokens helper."""
 
     def test_empty_string(self) -> None:
-        assert _estimate_token_count("") == 0
+        assert _approx_tokens("") == 0
 
     def test_short_string(self) -> None:
-        assert _estimate_token_count("hello") == 1  # 5 // 4 = 1
+        assert _approx_tokens("hello") == 1  # 5 // 4 = 1
 
     def test_longer_string(self) -> None:
-        assert _estimate_token_count("a" * 100) == 25  # 100 // 4 = 25
+        assert _approx_tokens("a" * 100) == 25  # 100 // 4 = 25
 
     def test_approximation_is_conservative(self) -> None:
         # For English text, actual token count is typically lower than char/4
         text = "The quick brown fox jumps over the lazy dog."
-        estimated = _estimate_token_count(text)
+        estimated = _approx_tokens(text)
         assert estimated > 0
         assert estimated == len(text) // 4
 
@@ -961,12 +944,10 @@ class TestParallelSummarization:
 
         # acall should have been awaited once per chunk
         assert mock_llm.acall.await_count == 3
-        # sync call should NOT have been used for chunk summarization
         mock_llm.call.assert_not_called()
 
-    def test_single_chunk_uses_sync_call(self) -> None:
-        """When there is only one chunk, summarize_messages should use
-        the sync llm.call path (no async overhead)."""
+    def test_single_chunk_uses_one_coroutine(self) -> None:
+        """One chunk still goes through _summarize_all."""
         messages: list[dict[str, Any]] = [
             {"role": "user", "content": "Short message"},
             {"role": "assistant", "content": "Short reply"},
@@ -974,7 +955,7 @@ class TestParallelSummarization:
 
         mock_llm = MagicMock()
         mock_llm.get_context_window_size.return_value = 100_000
-        mock_llm.call.return_value = "<summary>Short summary</summary>"
+        mock_llm.acall = AsyncMock(return_value="<summary>Short summary</summary>")
 
         summarize_messages(
             messages=messages,
@@ -983,7 +964,7 @@ class TestParallelSummarization:
 
         )
 
-        mock_llm.call.assert_called_once()
+        mock_llm.acall.assert_awaited_once()
 
     def test_parallel_results_preserve_order(self) -> None:
         """Summaries must appear in the same order as the original chunks,
@@ -1020,8 +1001,8 @@ class TestParallelSummarization:
         pos_c = summary_content.index("Summary-C")
         assert pos_a < pos_b < pos_c
 
-    def test_asummarize_chunks_returns_ordered_results(self) -> None:
-        """Direct test of the async helper _asummarize_chunks."""
+    def test_summarize_all_returns_ordered_results(self) -> None:
+        """Direct test of _summarize_all."""
         chunk_a: list[dict[str, Any]] = [{"role": "user", "content": "Chunk A"}]
         chunk_b: list[dict[str, Any]] = [{"role": "user", "content": "Chunk B"}]
 
@@ -1033,18 +1014,77 @@ class TestParallelSummarization:
             ]
         )
 
-        results = asyncio.run(
-            _asummarize_chunks(
-                chunks=[chunk_a, chunk_b],
-                llm=mock_llm,
-                callbacks=[],
-    
-            )
+        summarizer = SummarizeMessages()
+        summarizer.llm = mock_llm
+        summarizer.callbacks = []
+        results = summarizer._summarize_all(
+            chunks=[chunk_a, chunk_b], char_level_index=0
         )
 
-        assert len(results) == 2
-        assert results[0]["content"] == "Result A"
-        assert results[1]["content"] == "Result B"
+        assert results == ["Result A", "Result B"]
+
+    def test_summarize_all_retries_after_context_length_error(self) -> None:
+        """A chunk that overflows context is retried after tighter token estimation."""
+        chunk: list[dict[str, Any]] = [{"role": "user", "content": "x" * 800}]
+
+        mock_llm = MagicMock()
+        mock_llm.get_context_window_size.return_value = 100_000
+        mock_llm.acall = AsyncMock(
+            side_effect=[
+                LLMContextLengthExceededError("context length exceeded"),
+                "<summary>Recovered summary</summary>",
+            ]
+        )
+
+        summarizer = SummarizeMessages()
+        summarizer.llm = mock_llm
+        summarizer.callbacks = []
+        summarizer.verbose = False
+
+        results = summarizer._summarize_all(chunks=[chunk], char_level_index=0)
+
+        assert results == ["Recovered summary"]
+        assert mock_llm.acall.await_count == 2
+
+    def test_summarize_messages_retries_each_char_per_token_level(self) -> None:
+        """Context errors step through chunk levels 0, 1, and 2 before succeeding."""
+        messages: list[dict[str, Any]] = [{"role": "user", "content": "x" * 800}]
+        recorded_levels: list[int] = []
+        original_chunk = SummarizeMessages._chunk_messages
+
+        def tracking_chunk(
+            self: SummarizeMessages,
+            chunk_messages: list[dict[str, Any]],
+            max_tokens: int,
+            char_level_index: int = 0,
+        ) -> list[list[dict[str, Any]]]:
+            recorded_levels.append(char_level_index)
+            return original_chunk(
+                self, chunk_messages, max_tokens, char_level_index=char_level_index
+            )
+
+        mock_llm = MagicMock()
+        mock_llm.get_context_window_size.return_value = 100_000
+        mock_llm.acall = AsyncMock(
+            side_effect=[
+                LLMContextLengthExceededError("context length exceeded"),
+                LLMContextLengthExceededError("context length exceeded"),
+                "<summary>Final summary</summary>",
+            ]
+        )
+
+        with patch.object(SummarizeMessages, "_chunk_messages", tracking_chunk):
+            summarize_messages(
+                messages=messages,
+                llm=mock_llm,
+                callbacks=[],
+                verbose=False,
+            )
+
+        assert len(messages) == 1
+        assert "Final summary" in messages[0]["content"]
+        assert recorded_levels == [0, 1, 2]
+        assert mock_llm.acall.await_count == 3
 
     @patch("crewai.utilities.agent_utils.is_inside_event_loop", return_value=True)
     def test_works_inside_existing_event_loop(self, _mock_loop: Any) -> None:
@@ -1170,7 +1210,7 @@ class TestParallelSummarizationVCR:
         # Patch get_context_window_size to return 200 — forces multiple chunks
         with patch.object(type(llm), "get_context_window_size", return_value=200):
             non_system = [m for m in messages if m.get("role") != "system"]
-            chunks = _split_messages_into_chunks(non_system, max_tokens=200)
+            chunks = _chunk_messages(non_system, max_tokens=200)
             assert len(chunks) > 1, f"Expected multiple chunks, got {len(chunks)}"
 
             summarize_messages(

pyproject.toml

@@ -209,6 +209,9 @@ exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings =
 # global 3-day cutoff is now far later than that date, so the per-package pin only blocked the fix.
 # pyasn1 <0.6.4 has GHSA-8ppf-4f7h-5ppj and GHSA-hm4w-wwcw-mr6r; force 0.6.4+.
 # urllib3 <2.7.0 has GHSA-qccp-gfcp-xxvc (ProxyManager cross-origin redirect leaks Authorization/Cookie) and GHSA-mf9v-mfxr-j63j (streaming decompression-bomb bypass); force 2.7.0+.
+# urllib3 <2.8.0 has GHSA-8988-9cw3-xx77 (HTTPS proxy TLS configuration ignored), GHSA-gh4c-6fx4-qh6g (chunked
+# deflate infinite loop) and GHSA-vxq7-64xx-v4gw (unbounded chunk-size line); force 2.8.0+.
+# virtualenv <21.7.13 has PYSEC-2026-4011..4014 (unchecked seed wheels, pyvenv.cfg and activator injection); force 21.7.13+.
 # langsmith <0.8.18 has GHSA-3644-q5cj-c5c7 (public prompt manifest deserialization, SSRF/secret disclosure)
 # and GHSA-f4xh-w4cj-qxq8; force 0.8.18+.
 # authlib <1.6.12 has GHSA-jj8c-mmj3-mmgv (CSRF bypass in cache-based state storage) and PYSEC-2026-188.
@@ -259,6 +262,11 @@ exclude-newer-package = { msgpack = "2026-06-20T00:00:00Z", pydantic-settings =
 # fixed in 4.7.1. Declared as crewai-tools[snowflake] "snowflake-connector-python>=3.12.4",
 # which the lock resolved to 4.6.0.
 # soupsieve <=2.8.4 has GHSA-j934-xhv5-fg8f and GHSA-gjv8-xp57-g29c; force 2.9.2+.
+# litellm <1.88.6 has GHSA-3cv6-jpf6-8222 (CVE-2026-84377; authenticated SSRF and provider-credential
+# exfiltration via request-body routing parameters in the LiteLLM proxy server; crewAI uses litellm only as
+# a client SDK, but pip-audit flags the package). The floor lives in lib/crewai/pyproject.toml (the
+# crewai[litellm] extra, "litellm>=1.88.6,<2") rather than here, since crewai declares litellm directly.
+# 1.88.6 is older than the global 3-day cutoff, so no exclude-newer-package override is needed.
 # Keep OpenAI on the SDK range required by CrewAI when transitive dependencies
 # loosen or pin their own lower versions.
 override-dependencies = [
@@ -269,7 +277,8 @@ override-dependencies = [
     "pillow>=12.3.0",
     "langchain-core>=1.3.3,<2",
     "langchain-text-splitters>=1.1.2,<2",
-    "urllib3>=2.7.0",
+    "urllib3>=2.8.0",
+    "virtualenv>=21.7.13",
     "transformers>=5.4.0; python_version >= '3.10'",
     "cryptography>=50.0.0",
     "pypdf>=6.16.1,<7",