andre/crewAI
1
0
Fork 0
mirror of https://github.com/crewAIInc/crewAI.git synced 2026-07-01 05:08:12 +00:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

feat: adopt directory-based docs versioning with Edge channel

Browse Source 
Switch docs.crewai.com from navigation-only versioning (every version
selector entry rendered the same docs/<lang>/* source files) to
Mintlify's directory-based versioning so each version selector entry
renders its own snapshot. Add an "Edge" channel under docs/edge/<lang>/*
that always reflects main HEAD for unreleased work, eliminating
pre-release leakage onto frozen release labels. External links to
canonical /<lang>/* URLs are preserved via wildcard redirects that
always land on the current default version.

Layout:
- docs/edge/<lang>/*         rolling source (you edit here)
- docs/edge/enterprise-api.*.yaml
- docs/v<X.Y.Z>/<lang>/*     frozen, immutable snapshots
- docs/v<X.Y.Z>/enterprise-api.*.yaml
- docs/images/               shared, append-only
- docs/docs.json             nav + redirects

URLs follow the Mintlify-idiomatic shape: /edge/<lang>/<page> for
Edge, /v<X.Y.Z>/<lang>/<page> for every frozen snapshot. The wildcard
redirects /<lang>/:slug* -> /<default>/<lang>/:slug* keep stale links
working, and every freeze rewrites them (plus all per-section/per-page
redirects) so destinations always resolve to the current default
without depending on a second redirect hop.

Release flow integration (devtools release):
- New module crewai_devtools.docs_versioning.freeze() materialises
  docs/v<X.Y.Z>/ from docs/edge/, rewrites openapi: refs inside the
  snapshot, inserts the version into every language block in
  docs.json, and refreshes all redirect destinations.
- _update_docs_and_create_pr() in cli.py now calls that freeze during
  Phase 2 of devtools release. Edge changelogs are updated first (so
  the snapshot freeze picks them up), then the snapshot is staged
  alongside docs.json, branched as docs/freeze-v<X.Y.Z>, and the PR
  is titled [docs-freeze] docs: snapshot and changelog for v<X.Y.Z>
  — the title prefix the new CI guard reads.
- The PR still gates tag, GitHub release, PyPI publish, and the
  enterprise release as before; no new PRs are added.
- Pre-releases (1.X.YaN, 1.X.YbN, ...) skip the snapshot — they ride
  Edge — and the docs PR title omits the [docs-freeze] prefix.
- docs_check (AI-generated docs scaffolding) writes to
  docs/edge/<lang>/* so newly-generated unreleased docs land in Edge
  and never accidentally touch a frozen snapshot.

Migration scripts (one-shot):
- scripts/docs/freeze_historical_versions.py reconstructs all 16
  historical snapshots (v1.10.0 .. v1.14.7) from git tags via
  git archive | tar, rewriting openapi: MDX refs so each snapshot
  reads its own enterprise-api YAML rather than the live one.
- scripts/docs/prefix_version_paths.py one-shot-migrates docs.json:
  rewrites every page path in 16 versioned blocks to point under
  docs/v<X.Y.Z>/, inserts a new Edge entry per language, tags
  v1.14.7 as Latest (default), prunes pages whose target file
  doesn't exist in the snapshot (e.g. docs/ar/ didn't exist before
  v1.12.0), and writes the wildcard + per-section redirects.
- scripts/docs/freeze_current_edge.py is now a thin CLI wrapper
  around docs_versioning.freeze for manual one-off freezes (e.g.
  retroactively snapshotting a forgotten release).

CI guards (.github/workflows/docs-snapshots.yml):
- Frozen snapshots under docs/v[0-9]*/ are immutable; only PRs whose
  title contains [docs-freeze] (i.e. release-cut PRs generated by
  devtools release or the manual wrapper) may modify them.
- Images under docs/images/ are append-only since snapshots share a
  single image directory. Deleting or renaming an image breaks every
  historical snapshot that still references it.

Restored docs/images/crewai-otel-export.png from PR #3673; it was
deleted in PR #4908 but v1.10.0 / v1.10.1 snapshots still reference
it. Restoring instead of editing the snapshots preserves historical
rendering fidelity and validates the new append-only rule
retroactively.

Tests:
- lib/devtools/tests/test_docs_versioning.py covers the freeze: file
  copy, openapi rewrite, version insertion, default demotion, redirect
  upserts, per-section redirect rewriting, idempotency, and invalid
  inputs.

Verified locally with mintlify broken-links: 0 broken links across
the full site (Edge + 16 frozen versions, 4 locales).

AGENTS.md (repo root) is the contributor guide for the new model;
RELEASING.md is the release-cut runbook; README's Contribution
section links to both.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Lucas Gomide
2026-06-17 09:33:56 -03:00
parent 7bb9bc7e1a
commit 93dafe2637
15793 changed files with 3237032 additions and 16873 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
lib/devtools/README.md
View File

@@ -34,13 +34,18 @@ devtools release 1.10.3 --skip-enterprise # skip enterprise release phase
2. Runs `uv sync`
3. Creates version bump PR against main, polls until merged
4. Generates release notes (OpenAI) from commits since last release
5. Updates changelogs (en, pt-BR, ko) and docs version switcher
6. Creates docs PR against main, polls until merged
7. Tags main and creates GitHub release
8. Triggers PyPI publish workflow
9. Clones enterprise repo, bumps versions and `crewai[tools]` dep, runs `uv sync`
10. Creates enterprise bump PR, polls until merged
11. Tags and creates GitHub release on enterprise repo
5. Updates Edge changelogs (`docs/edge/{en,pt-BR,ko,ar}/changelog.mdx`)
6. Freezes `docs/edge/` into `docs/v<version>/`, registers the version in `docs.json`, and points the canonical `/<lang>/:slug*` redirects at the new default
7. Opens a `[docs-freeze]` PR against main, polls until merged
8. Tags main and creates GitHub release
9. Triggers PyPI publish workflow
10. Clones enterprise repo, bumps versions and `crewai[tools]` dep, runs `uv sync`
11. Creates enterprise bump PR, polls until merged
12. Tags and creates GitHub release on enterprise repo
> The `docs-snapshots` CI guard rejects writes under `docs/v*/` and deletions/renames in `docs/images/` unless the PR title starts with `[docs-freeze]`. The release CLI sets that prefix automatically; manual edits to a frozen snapshot need the same prefix to land.
>
> Pre-releases (e.g. `1.10.1b1`) skip the snapshot step — they ride Edge — and the docs PR title omits the `[docs-freeze]` prefix.
### `devtools bump <version>`

248
lib/devtools/src/crewai_devtools/cli.py
View File

@@ -22,6 +22,11 @@ from rich.prompt import Confirm
import tomlkit
from crewai_devtools.docs_check import docs_check
from crewai_devtools.docs_versioning import (
InvalidVersionError,
MissingEdgeSourcesError,
freeze as freeze_docs,
)
from crewai_devtools.prompts import RELEASE_NOTES_PROMPT, TRANSLATE_RELEASE_NOTES_PROMPT
@@ -390,56 +395,39 @@ def update_pyproject_dependencies(
def add_docs_version(docs_json_path: Path, version: str) -> bool:
"""Add a new version to the Mintlify docs.json versioning config.
"""Freeze Edge into a new snapshot and register the version in docs.json.
Copies the current default version's tabs into a new version entry,
sets the new version as default, and marks the previous default as
non-default. Operates on all languages.
Thin compatibility wrapper around :func:`crewai_devtools.docs_versioning.freeze`.
Materialises ``docs/v<version>/`` from ``docs/edge/`` (copies files, rewrites
``openapi:`` refs inside the snapshot), inserts a new ``vX.Y.Z`` entry into
every language's ``versions[]`` block just after Edge, marks it
default + ``Latest`` (demoting the prior default), and updates the wildcard
``/<lang>/:slug*`` redirects to point at the new version.
Skipped (returns False) for pre-release versions like ``1.10.1b1`` since
those don't get their own snapshot — pre-release docs stay on Edge.
Args:
docs_json_path: Path to docs/docs.json.
version: Version string (e.g., "1.10.1b1").
version: Version string (e.g., ``"1.10.1"``). Pre-releases are skipped.
Returns:
True if docs.json was updated, False otherwise.
True if docs.json was updated, False otherwise (missing file, missing
Edge sources, pre-release, or snapshot already up to date).
"""
import json
if not docs_json_path.exists():
return False
data = json.loads(docs_json_path.read_text())
version_label = f"v{version}"
updated = False
for lang in data.get("navigation", {}).get("languages", []):
versions = lang.get("versions", [])
if not versions:
continue
if any(v.get("version") == version_label for v in versions):
continue
default_version = next(
(v for v in versions if v.get("default")),
versions[0],
)
new_version = {
"version": version_label,
"default": True,
"tabs": default_version.get("tabs", []),
}
default_version.pop("default", None)
versions.insert(0, new_version)
updated = True
if not updated:
if _is_prerelease(version):
return False
docs_json_path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
return True
docs_root = docs_json_path.parent
try:
result = freeze_docs(version, docs_root)
except (InvalidVersionError, MissingEdgeSourcesError) as e:
console.print(f"[yellow]Warning:[/yellow] {e}")
return False
return result.docsjson_entries_inserted > 0 or result.redirects_upserted > 0
ChangelogLang = Literal["en", "pt-BR", "ko", "ar"]
@@ -1104,97 +1092,127 @@ def _update_docs_and_create_pr(
is_prerelease: bool,
dry_run: bool,
) -> str | None:
"""Update changelogs and docs version switcher, create PR if needed.
"""Update Edge changelogs, freeze a snapshot, and open the docs PR.
For a stable release this freezes ``docs/edge/`` into ``docs/v<version>/``
(after the Edge changelogs have been updated so the snapshot contains the
new entry), updates ``docs/docs.json`` to register the new version and
canonical-URL redirects, and opens a ``[docs-freeze]`` PR. The
``docs-snapshots`` CI guard recognises that title prefix and allows the
snapshot directory to land.
For a pre-release, only the Edge changelogs are touched (pre-releases don't
get a frozen snapshot — they ride Edge), and the PR title omits the
``[docs-freeze]`` prefix.
Returns:
The docs branch name if a PR was created, None otherwise.
"""
docs_json_path = cwd / "docs" / "docs.json"
edge_root = cwd / "docs" / "edge"
snapshot_path = cwd / "docs" / f"v{version}"
changelog_langs: list[ChangelogLang] = ["en", "pt-BR", "ko", "ar"]
if not dry_run:
docs_files_staged: list[str] = []
if dry_run:
for lang in changelog_langs:
cl_path = cwd / "docs" / lang / "changelog.mdx"
if lang == "en":
notes_for_lang = release_notes
else:
console.print(f"[dim]Translating release notes to {lang}...[/dim]")
notes_for_lang = translate_release_notes(
release_notes, lang, openai_client
)
if update_changelog(cl_path, version, notes_for_lang, lang=lang):
console.print(f"[green]✓[/green] Updated {cl_path.relative_to(cwd)}")
docs_files_staged.append(str(cl_path))
else:
console.print(
f"[yellow]Warning:[/yellow] Changelog not found at {cl_path.relative_to(cwd)}"
)
cl_path = edge_root / lang / "changelog.mdx"
translated = " (translated)" if lang != "en" else ""
console.print(
f"[dim][DRY RUN][/dim] Would update "
f"{cl_path.relative_to(cwd)}{translated}"
)
if not is_prerelease:
if add_docs_version(docs_json_path, version):
console.print(
f"[green]✓[/green] Added v{version} to docs version switcher"
)
docs_files_staged.append(str(docs_json_path))
else:
console.print(
f"[yellow]Warning:[/yellow] docs.json not found at {docs_json_path.relative_to(cwd)}"
)
if docs_files_staged:
docs_branch = f"docs/changelog-v{version}"
create_or_reset_branch(docs_branch)
for f in docs_files_staged:
run_command(["git", "add", f])
run_command(
[
"git",
"commit",
"-m",
f"docs: update changelog and version for v{version}",
]
console.print(
f"[dim][DRY RUN][/dim] Would freeze docs/edge -> "
f"{snapshot_path.relative_to(cwd)} and update docs.json + redirects"
)
console.print("[green]✓[/green] Committed docs updates")
run_command(["git", "push", "-u", "origin", docs_branch])
console.print(f"[green]✓[/green] Pushed branch {docs_branch}")
pr_url = run_command(
[
"gh",
"pr",
"create",
"--base",
"main",
"--title",
f"docs: update changelog and version for v{version}",
"--body",
"",
]
else:
console.print(
"[dim][DRY RUN][/dim] Skipping snapshot freeze (pre-release stays on Edge)"
)
console.print("[green]✓[/green] Created docs PR")
console.print(f"[cyan]PR URL:[/cyan] {pr_url}")
return docs_branch
prefix = "" if is_prerelease else "[docs-freeze] "
console.print(
f"[dim][DRY RUN][/dim] Would create branch docs/freeze-v{version}, "
f"open PR titled '{prefix}docs: snapshot and changelog for v{version}', "
"and wait for merge"
)
return None
docs_paths_staged: list[str] = []
# Step 1: update Edge changelogs first so the snapshot we freeze afterwards
# contains the new release's entry.
for lang in changelog_langs:
cl_path = cwd / "docs" / lang / "changelog.mdx"
translated = " (translated)" if lang != "en" else ""
console.print(
f"[dim][DRY RUN][/dim] Would update {cl_path.relative_to(cwd)}{translated}"
)
cl_path = edge_root / lang / "changelog.mdx"
if lang == "en":
notes_for_lang = release_notes
else:
console.print(f"[dim]Translating release notes to {lang}...[/dim]")
notes_for_lang = translate_release_notes(release_notes, lang, openai_client)
if update_changelog(cl_path, version, notes_for_lang, lang=lang):
console.print(f"[green]✓[/green] Updated {cl_path.relative_to(cwd)}")
docs_paths_staged.append(str(cl_path))
else:
console.print(
f"[yellow]Warning:[/yellow] Changelog not found at "
f"{cl_path.relative_to(cwd)}"
)
# Step 2: stable releases get a frozen snapshot + docs.json updates;
# pre-releases ride Edge so we only need the changelog edits.
is_freeze = False
if not is_prerelease:
console.print(
f"[dim][DRY RUN][/dim] Would add v{version} to docs version switcher"
)
else:
console.print("[dim][DRY RUN][/dim] Skipping docs version (pre-release)")
console.print(
f"[dim][DRY RUN][/dim] Would create branch docs/changelog-v{version}, PR, and wait for merge"
if add_docs_version(docs_json_path, version):
console.print(
f"[green]✓[/green] Froze docs/edge -> "
f"{snapshot_path.relative_to(cwd)} and updated docs.json + redirects"
)
docs_paths_staged.append(str(docs_json_path))
docs_paths_staged.append(str(snapshot_path))
is_freeze = True
else:
console.print(
f"[yellow]Warning:[/yellow] docs freeze did not modify "
f"{docs_json_path.relative_to(cwd)} "
"(missing file, missing Edge sources, or snapshot already current)"
)
if not docs_paths_staged:
return None
docs_branch = f"docs/freeze-v{version}"
create_or_reset_branch(docs_branch)
for path in docs_paths_staged:
run_command(["git", "add", path])
# The [docs-freeze] title prefix is what the docs-snapshots CI guard reads
# to allow writes under docs/v*/ and image deletions. Omit it for
# pre-releases since they don't touch frozen snapshots.
title_prefix = "[docs-freeze] " if is_freeze else ""
pr_title = f"{title_prefix}docs: snapshot and changelog for v{version}"
run_command(["git", "commit", "-m", pr_title])
console.print("[green]✓[/green] Committed docs updates")
run_command(["git", "push", "-u", "origin", docs_branch])
console.print(f"[green]✓[/green] Pushed branch {docs_branch}")
pr_url = run_command(
[
"gh",
"pr",
"create",
"--base",
"main",
"--title",
pr_title,
"--body",
"",
]
)
return None
console.print("[green]✓[/green] Created docs PR")
console.print(f"[cyan]PR URL:[/cyan] {pr_url}")
return docs_branch
def _create_tag_and_release(

15
lib/devtools/src/crewai_devtools/docs_check.py
View File

@@ -389,13 +389,18 @@ def docs_check(base: str, write: bool, dry_run: bool) -> None:
)
return
# Newly-generated docs are by definition unreleased, so they belong in the
# Edge channel (docs/edge/<lang>/...). Frozen snapshots under docs/v<X.Y.Z>/
# are immutable and CI rejects writes there outside [docs-freeze] PRs.
edge_root = docs_dir / "edge"
for action_item in analysis.actions:
if action_item.action not in ("create", "update") or not action_item.file:
continue
rel_path = action_item.file
en_path = (docs_dir / "en" / rel_path).resolve()
if not en_path.is_relative_to(docs_dir.resolve()):
en_path = (edge_root / "en" / rel_path).resolve()
if not en_path.is_relative_to(edge_root.resolve()):
console.print(f" [red]✗ Skipping unsafe path: {rel_path!r}[/red]")
continue
console.print(f"\n[bold]Processing:[/bold] {rel_path}")
@@ -452,10 +457,10 @@ def docs_check(base: str, write: bool, dry_run: bool) -> None:
if not content:
continue
resolved_docs = docs_dir.resolve()
resolved_edge = edge_root.resolve()
for lang in _TRANSLATION_LANGS:
lang_path = (docs_dir / lang / rel_path).resolve()
if not lang_path.is_relative_to(resolved_docs):
lang_path = (edge_root / lang / rel_path).resolve()
if not lang_path.is_relative_to(resolved_edge):
continue
with console.status(f" [cyan]Translating to {_LANGUAGE_NAMES[lang]}..."):

429
lib/devtools/src/crewai_devtools/docs_versioning.py Normal file
View File

@@ -0,0 +1,429 @@
"""Freeze the current Edge docs into a per-version snapshot.
Used by ``devtools release`` (and the standalone
``scripts/docs/freeze_current_edge.py`` wrapper) during the docs PR step, which
runs *before* the release tag is created and PyPI publish is triggered. Once
the docs PR merges, the site reflects the new release at ``/v<X.Y.Z>/...`` and
the canonical ``/<lang>/...`` URLs (kept stable for external links) start
redirecting to the new default version.
Layout assumptions (set up by ``scripts/docs/prefix_version_paths.py``):
- ``docs/edge/`` rolling source matching main HEAD
- ``en/``, ``pt-BR/``, ``ko/``, ``ar/``
- ``enterprise-api.*.yaml``
- ``docs/v<X.Y.Z>/`` frozen, immutable snapshots
- ``docs/docs.json`` Mintlify config: ``navigation`` + ``redirects``
A freeze does four things:
1. Copy ``docs/edge/*`` into ``docs/v<X.Y.Z>/``.
2. Rewrite ``openapi:`` MDX refs inside the snapshot to point at the snapshot's
own ``enterprise-api.*.yaml`` (otherwise frozen pages would render against
whatever YAML happens to be at docs root today).
3. Insert a new version entry into every language's ``versions[]`` block in
``docs.json``, place it just after Edge, mark it default + Latest, and demote
the prior default.
4. Update wildcard redirects in ``docs.json`` so ``/<lang>/:slug*`` lands on the
new default version.
Idempotent: re-running with a version that already has a snapshot directory
*and* a docs.json entry is a no-op.
"""
from __future__ import annotations
from collections.abc import Callable
import copy
from dataclasses import dataclass
import json
from pathlib import Path
import re
import shutil
from typing import Any, Final
VERSION_RE: Final[re.Pattern[str]] = re.compile(r"^\d+\.\d+\.\d+$")
VERSION_SLUG_RE: Final[re.Pattern[str]] = re.compile(r"^v\d+\.\d+\.\d+$")
EDGE_VERSION: Final[str] = "Edge"
EDGE_PREFIX: Final[str] = "edge"
LATEST_TAG: Final[str] = "Latest"
KNOWN_LOCALES: Final[tuple[str, ...]] = ("en", "pt-BR", "ko", "ar")
# Per-snapshot copies are sourced from docs/edge/<name>. The frozen layout
# under docs/v<tag>/ omits the ``edge/`` segment so its URLs are
# ``/v<tag>/<lang>/...``.
SNAPSHOT_PATHS: Final[tuple[str, ...]] = (
"en",
"pt-BR",
"ko",
"ar",
"enterprise-api.base.yaml",
"enterprise-api.en.yaml",
"enterprise-api.ko.yaml",
"enterprise-api.pt-BR.yaml",
)
PAGE_EXTENSIONS: Final[tuple[str, ...]] = (".mdx", ".md")
# Matches ``openapi: "/enterprise-api.en.yaml ..."``. The snapshot version of
# the MDX needs the path prefixed with ``v<tag>/`` so the page reads the
# frozen YAML rather than whichever YAML happens to live at docs root.
_OPENAPI_PATTERN: Final[re.Pattern[str]] = re.compile(
r'(openapi:\s*"\s*)/(enterprise-api\.[^"\s]+\.yaml)'
)
class InvalidVersionError(ValueError):
"""Raised when a freeze is requested with a non-X.Y.Z version string."""
class MissingEdgeSourcesError(RuntimeError):
"""Raised when ``docs/edge/`` is missing or has no snapshot paths."""
@dataclass(frozen=True)
class FreezeResult:
"""Structured outcome of a freeze, for callers that render their own UI."""
version_slug: str
snapshot_path: Path
files_copied: int
openapi_refs_rewritten: int
docsjson_entries_inserted: int
docsjson_entries_skipped: int
redirects_upserted: int
snapshot_already_existed: bool
def freeze(version: str, docs_root: Path) -> FreezeResult:
"""Freeze the current Edge into ``docs/v<version>/`` and update docs.json.
Args:
version: Release version as ``"X.Y.Z"`` (no leading ``v``).
docs_root: Path to the ``docs/`` directory.
Returns:
``FreezeResult`` summarising what changed.
Raises:
InvalidVersionError: ``version`` is not an X.Y.Z string.
MissingEdgeSourcesError: ``docs/edge/`` is absent or empty.
"""
if not VERSION_RE.match(version):
raise InvalidVersionError(f"{version!r} is not a valid X.Y.Z version string")
version_slug = f"v{version}"
target = docs_root / version_slug
docs_json = docs_root / "docs.json"
snapshot_already_existed = target.exists()
if snapshot_already_existed:
files_copied = 0
openapi_refs_rewritten = 0
else:
files_copied = _copy_snapshot(docs_root, target)
openapi_refs_rewritten = _rewrite_openapi_refs(target, version_slug)
inserted, skipped, redirects_upserted = _migrate_docs_json(docs_json, version_slug)
return FreezeResult(
version_slug=version_slug,
snapshot_path=target,
files_copied=files_copied,
openapi_refs_rewritten=openapi_refs_rewritten,
docsjson_entries_inserted=inserted,
docsjson_entries_skipped=skipped,
redirects_upserted=redirects_upserted,
snapshot_already_existed=snapshot_already_existed,
)
def _copy_snapshot(docs_root: Path, target: Path) -> int:
"""Copy Edge sources under ``docs/edge/`` into ``target``.
Returns the number of files copied (recursively across directories).
"""
edge_root = docs_root / EDGE_PREFIX
if not edge_root.is_dir():
raise MissingEdgeSourcesError(
f"Expected Edge sources under {edge_root}/. "
"Did you forget to migrate Edge into docs/edge/?"
)
target.mkdir(parents=True, exist_ok=True)
count = 0
for name in SNAPSHOT_PATHS:
src = edge_root / name
if not src.exists():
continue
dst = target / name
if src.is_dir():
shutil.copytree(src, dst)
count += sum(1 for p in dst.rglob("*") if p.is_file())
else:
shutil.copy2(src, dst)
count += 1
if count == 0:
raise MissingEdgeSourcesError(
f"docs/edge/ exists but contains none of {list(SNAPSHOT_PATHS)}"
)
return count
def _rewrite_openapi_refs(target: Path, version_slug: str) -> int:
"""Prefix every ``openapi:`` reference in the snapshot with the version."""
rewritten = 0
for mdx in target.rglob("*.mdx"):
text = mdx.read_text(encoding="utf-8")
new_text, n = _OPENAPI_PATTERN.subn(rf"\1/{version_slug}/\2", text)
if n:
mdx.write_text(new_text, encoding="utf-8")
rewritten += n
return rewritten
def _walk_pages(node: Any, transform: Callable[[str], str]) -> Any:
"""Recursively walk a nav subtree, applying ``transform`` to every leaf."""
if isinstance(node, str):
return transform(node)
if isinstance(node, list):
return [_walk_pages(item, transform) for item in node]
if isinstance(node, dict):
out = dict(node)
for key in ("pages", "tabs", "groups"):
if key in out:
out[key] = [_walk_pages(c, transform) for c in out[key]]
return out
return node
def _is_version_slug(value: str) -> bool:
return bool(VERSION_SLUG_RE.match(value))
def _previous_default(versions: list[dict]) -> dict | None:
"""Return the entry currently marked default (or the first versioned)."""
for v in versions:
if v.get("default") and _is_version_slug(v.get("version", "")):
return v
for v in versions:
if _is_version_slug(v.get("version", "")):
return v
return None
def _build_new_entry(
previous: dict, version_slug: str, locale: str, docs_root: Path
) -> dict | None:
"""Clone the previous default's nav into a new entry for ``version_slug``.
Page paths are rewritten from ``v<prev>/<locale>/...`` to
``v<new>/<locale>/...``. Paths that don't resolve to a file in the
snapshot are pruned and the now-empty groups/tabs cascade away. Returns
``None`` if the locale has no resolvable content under the snapshot (e.g.
a locale that wasn't present in Edge yet).
"""
new_entry = copy.deepcopy(previous)
new_entry["version"] = version_slug
new_entry["default"] = True
new_entry["tag"] = LATEST_TAG
old_prefix = re.compile(rf"^{re.escape(previous['version'])}/")
locale_prefix = f"{locale}/"
new_prefix = f"{version_slug}/"
def transform(page: str) -> str:
if page.startswith(new_prefix):
return page
rewritten = old_prefix.sub(new_prefix, page)
if rewritten != page:
return rewritten
if page.startswith(locale_prefix):
return f"{new_prefix}{page}"
return page
rewritten = _walk_pages(new_entry, transform)
pruned = _prune_missing_pages(rewritten, docs_root)
if not pruned or not pruned.get("tabs"):
return None
return pruned
def _prune_missing_pages(node: Any, docs_root: Path) -> Any:
"""Drop pages whose target file is missing; cascade-empty groups/tabs."""
if isinstance(node, str):
for ext in PAGE_EXTENSIONS:
if (docs_root / f"{node}{ext}").is_file():
return node
return None
if isinstance(node, list):
kept = [_prune_missing_pages(item, docs_root) for item in node]
return [k for k in kept if k is not None]
if isinstance(node, dict):
out: dict = {}
for key, value in node.items():
if key in {"pages", "tabs", "groups"}:
pruned = _prune_missing_pages(value, docs_root)
if pruned:
out[key] = pruned
else:
out[key] = value
if "pages" in node and not out.get("pages"):
return None
if "groups" in node and not out.get("groups"):
return None
if "tabs" in node and not out.get("tabs"):
return None
return out
return node
def _drop_latest_marker(entry: dict) -> dict:
out = dict(entry)
out.pop("default", None)
if out.get("tag") == LATEST_TAG:
out.pop("tag")
return out
def _update_redirects(data: dict, version_slug: str) -> int:
"""Make every redirect destination land on the current default version.
Two passes:
1. Upsert the wildcard ``/<locale>/:slug*`` -> ``/<version_slug>/<locale>/:slug*``
entries so stale canonical URLs (``/en/...``, ``/ko/...``, etc.) keep
resolving.
2. Rewrite the destination of any pre-existing redirect that lands on a
bare ``/<locale>/...`` or stale ``/v<old>/<locale>/...`` path so it
lands on the current default version directly. Mintlify's link checker
resolves each redirect independently and does not chain through them,
so a destination that depends on a second redirect counts as broken.
Returns the number of redirect entries that were inserted or modified.
"""
redirects = data.setdefault("redirects", [])
if not isinstance(redirects, list):
raise RuntimeError("docs.json 'redirects' is not a list")
upserted = 0
for locale in KNOWN_LOCALES:
source = f"/{locale}/:slug*"
destination = f"/{version_slug}/{locale}/:slug*"
existing = next(
(r for r in redirects if isinstance(r, dict) and r.get("source") == source),
None,
)
if existing is None:
redirects.append(
{"source": source, "destination": destination, "permanent": False}
)
upserted += 1
elif existing.get("destination") != destination:
existing["destination"] = destination
existing["permanent"] = False
upserted += 1
for entry in redirects:
if not isinstance(entry, dict):
continue
destination = entry.get("destination")
if not isinstance(destination, str):
continue
new_destination = _rewrite_destination_to_version(destination, version_slug)
if new_destination != destination:
entry["destination"] = new_destination
upserted += 1
return upserted
def _rewrite_destination_to_version(destination: str, version_slug: str) -> str:
"""Rewrite a redirect destination to land on ``version_slug`` directly.
Handles three shapes:
- ``/<locale>/...`` -> ``/<version_slug>/<locale>/...``
- ``/v<X.Y.Z>/<locale>/...`` -> ``/<version_slug>/<locale>/...``
- anything else -> unchanged
"""
if not destination.startswith("/"):
return destination
parts = destination.lstrip("/").split("/", 2)
if not parts:
return destination
head = parts[0]
if head in KNOWN_LOCALES:
return f"/{version_slug}/{destination.lstrip('/')}"
if VERSION_SLUG_RE.match(head) and len(parts) >= 2 and parts[1] in KNOWN_LOCALES:
if head == version_slug:
return destination
rest = "/".join(parts[1:])
return f"/{version_slug}/{rest}"
return destination
def _migrate_docs_json(docs_json: Path, version_slug: str) -> tuple[int, int, int]:
"""Insert a new versioned entry per language and refresh redirects."""
data = json.loads(docs_json.read_text(encoding="utf-8"))
docs_root = docs_json.parent
inserted = 0
skipped = 0
for block in data["navigation"]["languages"]:
locale = block["language"]
versions: list[dict] = block.get("versions", [])
if any(v.get("version") == version_slug for v in versions):
skipped += 1
continue
previous = _previous_default(versions)
if previous is None:
skipped += 1
continue
new_entry = _build_new_entry(previous, version_slug, locale, docs_root)
if new_entry is None:
# Locale has no resolvable content under the snapshot yet (e.g. a
# locale that didn't exist in Edge). Leave the block untouched.
skipped += 1
continue
updated: list[dict] = []
for v in versions:
if v.get("default") or v.get("tag") == LATEST_TAG:
updated.append(_drop_latest_marker(v))
else:
updated.append(v)
# Insert the new versioned entry just after Edge so the version selector
# shows: Edge, vNEW (default/Latest), vPREV, vPREV-1, ...
edge_idx = next(
(i for i, v in enumerate(updated) if v.get("version") == EDGE_VERSION),
-1,
)
insert_at = edge_idx + 1 if edge_idx >= 0 else 0
updated.insert(insert_at, new_entry)
block["versions"] = updated
inserted += 1
redirects_upserted = _update_redirects(data, version_slug)
docs_json.write_text(
json.dumps(data, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
return inserted, skipped, redirects_upserted

226
lib/devtools/tests/test_docs_versioning.py Normal file
View File

@@ -0,0 +1,226 @@
"""Tests for the Edge -> snapshot freeze used by the docs PR step."""
from __future__ import annotations
import json
from pathlib import Path
from crewai_devtools.docs_versioning import (
InvalidVersionError,
MissingEdgeSourcesError,
freeze,
)
import pytest
def _build_docs_root(tmp_path: Path) -> Path:
"""Build a minimal docs/ tree with one previous snapshot + Edge + docs.json.
The shape mirrors the real repo: an Edge directory with per-locale folders
and YAMLs, one previously-frozen v1.14.7 snapshot, and a docs.json with
Edge + the previous default in the version selector plus canonical-URL
wildcard redirects.
"""
docs = tmp_path / "docs"
# Edge sources (what the release will freeze).
edge_en = docs / "edge" / "en"
edge_en.mkdir(parents=True)
(edge_en / "introduction.mdx").write_text("# Intro (Edge)\n")
(edge_en / "changelog.mdx").write_text("---\ntitle: Changelog\n---\n")
(edge_en / "api.mdx").write_text(
'---\nopenapi: "/enterprise-api.en.yaml GET /foo"\n---\n'
)
(docs / "edge" / "enterprise-api.en.yaml").write_text("openapi: 3.0.0\n")
# A pre-existing frozen snapshot to clone the nav structure from.
snap_en = docs / "v1.14.7" / "en"
snap_en.mkdir(parents=True)
(snap_en / "introduction.mdx").write_text("# Intro (1.14.7)\n")
(snap_en / "changelog.mdx").write_text("---\ntitle: Changelog\n---\n")
(snap_en / "api.mdx").write_text(
'---\nopenapi: "/v1.14.7/enterprise-api.en.yaml GET /foo"\n---\n'
)
docs_json = {
"navigation": {
"languages": [
{
"language": "en",
"versions": [
{
"version": "Edge",
"tag": "Edge",
"tabs": [
{
"tab": "Guides",
"pages": [
"edge/en/introduction",
"edge/en/changelog",
"edge/en/api",
],
}
],
},
{
"version": "v1.14.7",
"default": True,
"tag": "Latest",
"tabs": [
{
"tab": "Guides",
"pages": [
"v1.14.7/en/introduction",
"v1.14.7/en/changelog",
"v1.14.7/en/api",
],
}
],
},
],
}
]
},
"redirects": [
{
"source": "/en/:slug*",
"destination": "/v1.14.7/en/:slug*",
"permanent": False,
}
],
}
(docs / "docs.json").write_text(json.dumps(docs_json, indent=2) + "\n")
return docs
class TestFreeze:
def test_copies_edge_files_into_snapshot(self, tmp_path: Path) -> None:
docs = _build_docs_root(tmp_path)
result = freeze("1.15.0", docs)
snapshot = docs / "v1.15.0"
assert snapshot.is_dir()
assert (snapshot / "en" / "introduction.mdx").read_text() == "# Intro (Edge)\n"
assert (snapshot / "enterprise-api.en.yaml").read_text() == "openapi: 3.0.0\n"
assert result.snapshot_path == snapshot
assert result.files_copied >= 4
assert result.snapshot_already_existed is False
def test_rewrites_openapi_refs_to_snapshot_yaml(self, tmp_path: Path) -> None:
docs = _build_docs_root(tmp_path)
result = freeze("1.15.0", docs)
frozen_api = (docs / "v1.15.0" / "en" / "api.mdx").read_text()
assert "/v1.15.0/enterprise-api.en.yaml" in frozen_api
# The Edge source must NOT be rewritten — it stays generic so the next
# release freezes pick up the same edit.
edge_api = (docs / "edge" / "en" / "api.mdx").read_text()
assert "/v1.15.0/" not in edge_api
assert "/enterprise-api.en.yaml" in edge_api
assert result.openapi_refs_rewritten >= 1
def test_inserts_version_after_edge_and_demotes_previous_default(
self, tmp_path: Path
) -> None:
docs = _build_docs_root(tmp_path)
freeze("1.15.0", docs)
data = json.loads((docs / "docs.json").read_text())
versions = data["navigation"]["languages"][0]["versions"]
labels = [v["version"] for v in versions]
assert labels == ["Edge", "v1.15.0", "v1.14.7"]
new_entry = versions[1]
assert new_entry["default"] is True
assert new_entry["tag"] == "Latest"
# Page paths in the new entry must point at the new snapshot.
page_strs = [
p for tab in new_entry["tabs"] for p in tab["pages"] if isinstance(p, str)
]
assert all(p.startswith("v1.15.0/en/") for p in page_strs)
previous = versions[2]
assert "default" not in previous
assert previous.get("tag") != "Latest"
def test_updates_canonical_url_redirect_to_new_default(
self, tmp_path: Path
) -> None:
docs = _build_docs_root(tmp_path)
result = freeze("1.15.0", docs)
data = json.loads((docs / "docs.json").read_text())
en_redirect = next(r for r in data["redirects"] if r["source"] == "/en/:slug*")
assert en_redirect["destination"] == "/v1.15.0/en/:slug*"
assert en_redirect["permanent"] is False
assert result.redirects_upserted >= 1
def test_rewrites_stale_per_section_redirects_to_new_default(
self, tmp_path: Path
) -> None:
# docs.json carries pre-existing per-section/per-page redirects whose
# destinations point at /<locale>/... or /v<prev>/<locale>/...; those
# need to land on the current default version directly because
# Mintlify's link-checker doesn't follow redirect chains.
docs = _build_docs_root(tmp_path)
data = json.loads((docs / "docs.json").read_text())
data["redirects"].extend(
[
{"source": "/concepts/:path*", "destination": "/en/concepts/:path*"},
{
"source": "/api-reference/:path*",
"destination": "/v1.14.7/en/api-reference/:path*",
},
{"source": "/introduction", "destination": "/en/introduction"},
{"source": "/external", "destination": "https://example.com/"},
]
)
(docs / "docs.json").write_text(json.dumps(data, indent=2) + "\n")
freeze("1.15.0", docs)
data = json.loads((docs / "docs.json").read_text())
by_source = {r["source"]: r["destination"] for r in data["redirects"]}
assert by_source["/concepts/:path*"] == "/v1.15.0/en/concepts/:path*"
assert by_source["/api-reference/:path*"] == "/v1.15.0/en/api-reference/:path*"
assert by_source["/introduction"] == "/v1.15.0/en/introduction"
# Absolute URLs and other non-locale destinations are left alone.
assert by_source["/external"] == "https://example.com/"
def test_is_idempotent_when_snapshot_already_exists(self, tmp_path: Path) -> None:
docs = _build_docs_root(tmp_path)
freeze("1.15.0", docs)
before = (docs / "docs.json").read_text()
result = freeze("1.15.0", docs)
assert result.snapshot_already_existed is True
assert result.files_copied == 0
assert result.openapi_refs_rewritten == 0
# docs.json shape doesn't change on the second run because the version
# is already registered.
assert result.docsjson_entries_inserted == 0
assert (docs / "docs.json").read_text() == before
def test_rejects_invalid_version_string(self, tmp_path: Path) -> None:
docs = _build_docs_root(tmp_path)
with pytest.raises(InvalidVersionError):
freeze("v1.15.0", docs)
with pytest.raises(InvalidVersionError):
freeze("1.15", docs)
with pytest.raises(InvalidVersionError):
freeze("1.15.0a1", docs)
def test_rejects_missing_edge_directory(self, tmp_path: Path) -> None:
docs = _build_docs_root(tmp_path)
import shutil
shutil.rmtree(docs / "edge")
with pytest.raises(MissingEdgeSourcesError):
freeze("1.15.0", docs)