Compare commits

...

2 Commits

Author SHA1 Message Date
Lorenze Jay
cf6c32e79b Merge branch 'main' into lorenze/imp/issue-driven-pull-requests 2026-08-04 12:32:26 -07:00
lorenzejay
66ce0c536c docs: define issue-first contribution workflow 2026-08-04 10:48:49 -07:00
9 changed files with 783 additions and 6 deletions

View File

@@ -42,6 +42,40 @@ Documentation lives in `docs/` with translations under `docs/{en,ar,ko,pt-BR}/`.
## Development Workflow
### Issue-First Contributions
CrewAI is piloting an issue-first contribution workflow. The issue is where
contributors and maintainers agree on the problem and intended solution before
implementation begins.
Issues move through the following lifecycle:
```
state:inbox -> state:design -> state:ready -> state:in-progress -> state:verification -> Done
```
Maintainers apply `state:ready` after the issue has a clear problem statement,
agreed scope and non-goals, acceptance criteria, and a verification plan. A
pull request should not begin before its issue is ready unless a maintainer has
approved an exception.
Every new pull request must reference exactly one ready issue in its description:
```
Implements #123
```
Use `Implements`, not `Closes`, `Fixes`, or `Resolves`. GitHub's closing
keywords close the issue as soon as the pull request merges, but issue-first
contributions remain open in `state:verification` until a human confirms the
result.
The issue gate currently runs in **observe mode**. It reports whether a pull
request would satisfy this policy, but it does not block or close pull requests.
Maintainers can apply `issue-gate:override` for security work, release
automation, urgent fixes, or another documented exception. Pull requests that
predate the pilot can be marked `policy:legacy`.
### Branching
Create a branch off `main` using the conventional commit type:
@@ -103,7 +137,9 @@ chore(deps): bump pydantic to 2.11
- Keep PRs focused — avoid bundling unrelated changes
- PRs over 500 lines are labeled `size/XL` automatically
- Title must follow the same conventional commit format
- Link related issues where applicable
- Reference exactly one ready issue with `Implements #<issue-number>`
- Explain how the implementation meets the issue's acceptance criteria
- Include the automated and manual verification performed
## Testing

56
.github/ISSUE_GATE.md vendored Normal file
View File

@@ -0,0 +1,56 @@
# Issue Gate Pilot
The issue gate checks whether a pull request references exactly one open issue
with the `state:ready` label. It runs on `pull_request_target`, checks out only
the trusted default branch, and never fetches or executes pull request code.
## Repository setup
Create these labels before activating the pilot:
- Issue lifecycle: `state:inbox`, `state:design`, `state:ready`,
`state:in-progress`, and `state:verification`
- Gate results: `issue-gate:passed`, `issue-gate:exempt`, and
`needs-ready-issue`
- Maintainer exceptions: `issue-gate:override` and `policy:legacy`
Configure these repository variables:
| Variable | Required | Value |
| --- | --- | --- |
| `ISSUE_GATE_MODE` | No | `observe` (default), `block`, or `close` |
| `ISSUE_GATE_CUTOFF` | Yes to activate | ISO 8601 timestamp for the first PR covered by the pilot |
| `ISSUE_GATE_READY_LABEL` | No | Defaults to `state:ready` |
Without `ISSUE_GATE_CUTOFF`, the workflow uses a future cutoff and treats every
pull request as legacy. This makes the workflow inert until maintainers choose
the activation time.
## Modes
- `observe`: Invalid pull requests receive a successful `Issue gate` status,
an explanatory comment, and the `needs-ready-issue` label.
- `block`: Invalid pull requests receive a failing status. Add `Issue gate` to
the `main` ruleset's required checks only after observation is complete.
- `close`: Invalid pull requests receive a failing status and are closed after
the comment is posted.
The script refuses to enter `block` or `close` mode without a configured cutoff.
Pull requests created by supported dependency automation, or labeled
`issue-gate:override` or `policy:legacy`, remain exempt.
## Testing the pilot
Run the `Issue Gate` workflow manually with a pull request number, or edit a
pull request description to trigger it again. Test at least these cases before
changing modes:
1. No implementation issue
2. A nonexistent, closed, or non-ready issue
3. More than one implementation issue
4. One open issue labeled `state:ready`
5. A pull request before the cutoff
6. A pull request with `issue-gate:override`
Keep the gate in `observe` mode until the sample contains at least 20 new pull
requests with no false positives.

View File

@@ -1,7 +1,7 @@
name: Bug report
description: Create a report to help us improve CrewAI
title: "[BUG]"
labels: ["bug"]
labels: ["bug", "state:inbox"]
assignees: []
body:
- type: textarea
@@ -30,6 +30,13 @@ body:
description: A clear and concise description of what you expected to happen.
validations:
required: true
- type: textarea
id: impact
attributes:
label: Impact
description: Describe who is affected and how this problem affects their work.
validations:
required: true
- type: textarea
id: screenshots-code
attributes:
@@ -106,6 +113,14 @@ body:
description: Have a solution in mind? Please suggest it here, or write "None".
validations:
required: true
- type: textarea
id: verification-plan
attributes:
label: Verification plan
description: Describe how a maintainer can confirm that a future fix resolves the problem.
placeholder: Include a minimal reproduction, expected assertions, or a manual verification procedure.
validations:
required: true
- type: textarea
id: additional-context
attributes:

View File

@@ -1,7 +1,7 @@
name: Feature request
description: Suggest a new feature for CrewAI
title: "[FEATURE]"
labels: ["feature-request"]
labels: ["feature-request", "state:inbox"]
assignees: []
body:
- type: markdown
@@ -26,8 +26,8 @@ body:
- type: textarea
id: problem
attributes:
label: Is your feature request related to a an existing bug? Please link it here.
description: A link to the bug or NA if not related to an existing bug.
label: Problem statement
description: Describe the user problem or unmet need without prescribing an implementation.
validations:
required: true
- type: textarea
@@ -44,6 +44,30 @@ body:
description: A clear and concise description of any alternative solutions or features you've considered.
validations:
required: false
- type: textarea
id: scope
attributes:
label: Proposed scope and non-goals
description: Describe what should be included and what should intentionally remain out of scope.
validations:
required: true
- type: textarea
id: acceptance-criteria
attributes:
label: Acceptance criteria
description: List the observable outcomes that would make this request complete.
placeholder: |
- [ ] A user can ...
- [ ] Existing behavior remains ...
validations:
required: true
- type: textarea
id: verification-plan
attributes:
label: Verification plan
description: Describe the automated or manual checks that should confirm the feature works.
validations:
required: true
- type: textarea
id: context
attributes:
@@ -62,4 +86,4 @@ body:
- I can test the feature once it's implemented
- No, I'm just suggesting the idea
validations:
required: true
required: true

24
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@@ -0,0 +1,24 @@
## Ready issue
Implements #<issue-number>
<!--
Replace the placeholder above with exactly one GitHub issue that has the
`state:ready` label. Use "Implements", not "Closes", "Fixes", or "Resolves":
the issue remains open for human verification after this PR is merged.
-->
## Summary
<!-- Explain the solution and how it follows the design agreed in the issue. -->
## Verification
<!-- List the automated and manual checks used to verify the change. -->
- [ ] Tests added or updated for the changed behavior
- [ ] Relevant tests and quality checks pass locally
## Additional context
<!-- Include screenshots, compatibility notes, follow-up work, or "None". -->

386
.github/scripts/issue-gate.mjs vendored Normal file
View File

@@ -0,0 +1,386 @@
import { readFile } from "node:fs/promises";
import { pathToFileURL } from "node:url";
const COMMENT_MARKER = "<!-- crewai-issue-gate -->";
const GATE_LABELS = [
"issue-gate:passed",
"issue-gate:exempt",
"needs-ready-issue",
];
const EXEMPT_LABELS = new Set(["issue-gate:override", "policy:legacy"]);
const EXEMPT_ACTORS = new Set([
"dependabot[bot]",
"github-actions[bot]",
"renovate[bot]",
]);
const VALID_MODES = new Set(["observe", "block", "close"]);
class ApiError extends Error {
constructor(status, message) {
super(message);
this.name = "ApiError";
this.status = status;
}
}
/** Return unique issue references declared with `Implements #123`. */
export function parseIssueReferences(body, defaultRepository) {
const references = [];
const seen = new Set();
const pattern =
/^[\t ]*Implements[\t ]+(?:(?<repository>[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+))?#(?<number>\d+)\b/gim;
for (const match of (body ?? "").matchAll(pattern)) {
const repository = match.groups.repository ?? defaultRepository;
const number = Number.parseInt(match.groups.number, 10);
const key = `${repository.toLowerCase()}#${number}`;
if (!seen.has(key)) {
references.push({ repository, number });
seen.add(key);
}
}
return references;
}
function labelNames(item) {
return new Set((item.labels ?? []).map((label) => label.name.toLowerCase()));
}
function exemptionReason(pullRequest, cutoff) {
const labels = labelNames(pullRequest);
for (const label of EXEMPT_LABELS) {
if (labels.has(label)) {
return {
kind: label === "policy:legacy" ? "legacy" : "override",
reason: `the pull request has the \`${label}\` exemption label`,
};
}
}
const actor = pullRequest.user?.login?.toLowerCase();
if (EXEMPT_ACTORS.has(actor)) {
return {
kind: "automation",
reason: `\`${pullRequest.user.login}\` is an exempt automation account`,
};
}
if (cutoff) {
const cutoffTime = Date.parse(cutoff);
if (Number.isNaN(cutoffTime)) {
throw new Error(`ISSUE_GATE_CUTOFF is not a valid date: ${cutoff}`);
}
const createdTime = Date.parse(pullRequest.created_at);
if (Number.isNaN(createdTime)) {
throw new Error(`Pull request has an invalid creation date: ${pullRequest.created_at}`);
}
if (createdTime < cutoffTime) {
return {
kind: "legacy",
reason: `the pull request predates the pilot cutoff (${cutoff})`,
};
}
}
return null;
}
/** Evaluate a pull request without mutating GitHub state. */
export async function evaluatePullRequest({
pullRequest,
repository,
cutoff = "",
readyLabel = "state:ready",
getIssue,
}) {
const exemption = exemptionReason(pullRequest, cutoff);
if (exemption) {
return {
ok: true,
exempt: true,
exemption: exemption.kind,
reason: exemption.reason,
};
}
const references = parseIssueReferences(pullRequest.body, repository);
if (references.length === 0) {
return {
ok: false,
exempt: false,
reason: "the description does not contain `Implements #<issue-number>`",
};
}
if (references.length > 1) {
return {
ok: false,
exempt: false,
reason: "the description references more than one implementation issue",
};
}
const reference = references[0];
if (reference.repository.toLowerCase() !== repository.toLowerCase()) {
return {
ok: false,
exempt: false,
reason: `the implementation issue must belong to \`${repository}\``,
};
}
const issue = await getIssue(reference.number);
if (!issue || issue.pull_request) {
return {
ok: false,
exempt: false,
reason: `#${reference.number} is not an issue in \`${repository}\``,
};
}
if (issue.state !== "open") {
return {
ok: false,
exempt: false,
issueNumber: reference.number,
reason: `issue #${reference.number} is not open`,
};
}
if (!labelNames(issue).has(readyLabel.toLowerCase())) {
return {
ok: false,
exempt: false,
issueNumber: reference.number,
reason: `issue #${reference.number} does not have the \`${readyLabel}\` label`,
};
}
return {
ok: true,
exempt: false,
issueNumber: reference.number,
reason: `issue #${reference.number} is open and ready for implementation`,
};
}
async function githubApi(token, method, endpoint, body) {
const response = await fetch(`https://api.github.com${endpoint}`, {
method,
headers: {
Accept: "application/vnd.github+json",
Authorization: `Bearer ${token}`,
"User-Agent": "crewai-issue-gate",
"X-GitHub-Api-Version": "2022-11-28",
},
body: body === undefined ? undefined : JSON.stringify(body),
});
const responseText = await response.text();
const responseBody = responseText ? JSON.parse(responseText) : null;
if (!response.ok) {
throw new ApiError(
response.status,
`${method} ${endpoint} failed (${response.status}): ${responseBody?.message ?? responseText}`,
);
}
return responseBody;
}
function renderComment(result, mode) {
const modeExplanation =
mode === "observe"
? "The gate is in **observe mode**, so this result does not block or close the pull request."
: `The gate is in **${mode} mode**.`;
let outcome;
if (result.exempt) {
outcome = `This pull request is exempt because ${result.reason}.`;
} else if (result.ok) {
outcome = `This pull request passes: ${result.reason}.`;
} else {
outcome = `This pull request would be rejected because ${result.reason}.`;
}
return `${COMMENT_MARKER}\n### Issue gate\n\n${outcome}\n\n${modeExplanation}\n\nTo satisfy the pilot policy, describe exactly one ready issue using \`Implements #123\`. After the issue becomes ready, edit the pull request description or ask a maintainer to re-run the Issue Gate workflow.`;
}
async function updateGateComment({ token, repository, pullNumber, result, mode }) {
const [owner, repo] = repository.split("/");
const comments = await githubApi(
token,
"GET",
`/repos/${owner}/${repo}/issues/${pullNumber}/comments?per_page=100`,
);
const existing = comments.find(
(comment) => comment.user?.type === "Bot" && comment.body?.includes(COMMENT_MARKER),
);
if (!existing && result.ok) {
return;
}
const body = renderComment(result, mode);
if (existing) {
await githubApi(
token,
"PATCH",
`/repos/${owner}/${repo}/issues/comments/${existing.id}`,
{ body },
);
} else {
await githubApi(
token,
"POST",
`/repos/${owner}/${repo}/issues/${pullNumber}/comments`,
{ body },
);
}
}
async function updateGateLabel({ token, repository, pullRequest, result }) {
const [owner, repo] = repository.split("/");
const desiredLabel = result.exempt
? "issue-gate:exempt"
: result.ok
? "issue-gate:passed"
: "needs-ready-issue";
const currentLabels = labelNames(pullRequest);
for (const label of GATE_LABELS) {
if (label !== desiredLabel && currentLabels.has(label)) {
try {
await githubApi(
token,
"DELETE",
`/repos/${owner}/${repo}/issues/${pullRequest.number}/labels/${encodeURIComponent(label)}`,
);
} catch (error) {
if (!(error instanceof ApiError) || error.status !== 404) {
throw error;
}
}
}
}
if (!currentLabels.has(desiredLabel)) {
try {
await githubApi(
token,
"POST",
`/repos/${owner}/${repo}/issues/${pullRequest.number}/labels`,
{ labels: [desiredLabel] },
);
} catch (error) {
if (error instanceof ApiError && [404, 422].includes(error.status)) {
console.warn(`Could not apply missing label \`${desiredLabel}\`; create the pilot labels first.`);
return;
}
throw error;
}
}
}
async function publishStatus({ token, repository, pullRequest, result, mode }) {
const [owner, repo] = repository.split("/");
const shouldPass = mode === "observe" || result.ok;
const prefix = mode === "observe" && !result.ok ? "Observe: would fail" : result.ok ? "Pass" : "Fail";
const description = `${prefix}${result.reason}`.slice(0, 140);
await githubApi(
token,
"POST",
`/repos/${owner}/${repo}/statuses/${pullRequest.head.sha}`,
{
state: shouldPass ? "success" : "failure",
context: "Issue gate",
description,
target_url: `https://github.com/${repository}/pull/${pullRequest.number}`,
},
);
}
async function run() {
const token = process.env.GITHUB_TOKEN;
const repository = process.env.GITHUB_REPOSITORY;
const eventPath = process.env.GITHUB_EVENT_PATH;
const pullNumber = Number.parseInt(process.env.PR_NUMBER, 10);
const mode = (process.env.ISSUE_GATE_MODE || "observe").toLowerCase();
const cutoff = process.env.ISSUE_GATE_CUTOFF || "";
const readyLabel = process.env.ISSUE_GATE_READY_LABEL || "state:ready";
if (!token || !repository || !eventPath || !Number.isInteger(pullNumber)) {
throw new Error("GITHUB_TOKEN, GITHUB_REPOSITORY, GITHUB_EVENT_PATH, and PR_NUMBER are required");
}
if (!VALID_MODES.has(mode)) {
throw new Error(`ISSUE_GATE_MODE must be one of: ${[...VALID_MODES].join(", ")}`);
}
if (mode !== "observe" && !cutoff) {
throw new Error("ISSUE_GATE_CUTOFF is required before enabling block or close mode");
}
const [owner, repo] = repository.split("/");
const event = JSON.parse(await readFile(eventPath, "utf8"));
const pullRequest =
event.pull_request ??
(await githubApi(token, "GET", `/repos/${owner}/${repo}/pulls/${pullNumber}`));
const result = await evaluatePullRequest({
pullRequest,
repository,
cutoff,
readyLabel,
getIssue: async (issueNumber) => {
try {
return await githubApi(
token,
"GET",
`/repos/${owner}/${repo}/issues/${issueNumber}`,
);
} catch (error) {
if (error instanceof ApiError && error.status === 404) {
return null;
}
throw error;
}
},
});
await publishStatus({ token, repository, pullRequest, result, mode });
if (result.exemption !== "legacy") {
await updateGateLabel({ token, repository, pullRequest, result });
await updateGateComment({
token,
repository,
pullNumber: pullRequest.number,
result,
mode,
});
}
if (mode === "close" && !result.ok) {
await githubApi(
token,
"PATCH",
`/repos/${owner}/${repo}/pulls/${pullRequest.number}`,
{ state: "closed" },
);
}
console.log(JSON.stringify({ mode, pullNumber: pullRequest.number, ...result }));
if (mode !== "observe" && !result.ok) {
process.exitCode = 1;
}
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((error) => {
console.error(error);
process.exitCode = 1;
});
}

166
.github/scripts/issue-gate.test.mjs vendored Normal file
View File

@@ -0,0 +1,166 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
evaluatePullRequest,
parseIssueReferences,
} from "./issue-gate.mjs";
const repository = "crewAIInc/crewAI";
function pullRequest(overrides = {}) {
return {
body: "Implements #123",
created_at: "2026-08-12T00:00:00Z",
labels: [],
user: { login: "contributor" },
...overrides,
};
}
function issue(overrides = {}) {
return {
number: 123,
state: "open",
labels: [{ name: "state:ready" }],
...overrides,
};
}
async function evaluate(pullOverrides = {}, issueOverrides = {}) {
return evaluatePullRequest({
pullRequest: pullRequest(pullOverrides),
repository,
cutoff: "2026-08-11T00:00:00Z",
getIssue: async () => issue(issueOverrides),
});
}
test("parses shorthand and full repository references", () => {
assert.deepEqual(
parseIssueReferences(
"Implements #123\nImplements crewAIInc/crewAI#456",
repository,
),
[
{ repository, number: 123 },
{ repository, number: 456 },
],
);
});
test("ignores GitHub closing keywords and template placeholders", () => {
assert.deepEqual(
parseIssueReferences(
"Fixes #123\nCloses #456\nImplements #<issue-number>",
repository,
),
[],
);
});
test("deduplicates repeated references", () => {
assert.deepEqual(
parseIssueReferences("Implements #123\nimplements #123", repository),
[{ repository, number: 123 }],
);
});
test("passes an open ready issue", async () => {
const result = await evaluate();
assert.equal(result.ok, true);
assert.equal(result.exempt, false);
assert.equal(result.issueNumber, 123);
});
test("rejects a missing issue reference", async () => {
const result = await evaluate({ body: "No issue yet" });
assert.equal(result.ok, false);
assert.match(result.reason, /does not contain/);
});
test("rejects more than one implementation issue", async () => {
const result = await evaluate({
body: "Implements #123\nImplements #456",
});
assert.equal(result.ok, false);
assert.match(result.reason, /more than one/);
});
test("rejects an issue in another repository", async () => {
const result = await evaluate({
body: "Implements another/project#123",
});
assert.equal(result.ok, false);
assert.match(result.reason, /must belong/);
});
test("rejects a pull request reference masquerading as an issue", async () => {
const result = await evaluate({}, { pull_request: { url: "example" } });
assert.equal(result.ok, false);
assert.match(result.reason, /is not an issue/);
});
test("rejects a closed issue", async () => {
const result = await evaluate({}, { state: "closed" });
assert.equal(result.ok, false);
assert.match(result.reason, /is not open/);
});
test("rejects an issue without the ready label", async () => {
const result = await evaluate({}, { labels: [{ name: "state:design" }] });
assert.equal(result.ok, false);
assert.match(result.reason, /does not have/);
});
test("exempts pull requests created before the cutoff", async () => {
const result = await evaluate({
body: null,
created_at: "2026-08-10T23:59:59Z",
});
assert.equal(result.ok, true);
assert.equal(result.exempt, true);
assert.match(result.reason, /predates/);
});
test("exempts pull requests with an override label", async () => {
const result = await evaluate({
body: null,
labels: [{ name: "issue-gate:override" }],
});
assert.equal(result.ok, true);
assert.equal(result.exempt, true);
assert.match(result.reason, /override/);
});
test("exempts supported automation accounts", async () => {
const result = await evaluate({
body: null,
user: { login: "dependabot[bot]" },
});
assert.equal(result.ok, true);
assert.equal(result.exempt, true);
assert.match(result.reason, /automation account/);
});
test("fails fast for an invalid cutoff", async () => {
await assert.rejects(
evaluatePullRequest({
pullRequest: pullRequest(),
repository,
cutoff: "not-a-date",
getIssue: async () => issue(),
}),
/not a valid date/,
);
});

24
.github/workflows/issue-gate-tests.yml vendored Normal file
View File

@@ -0,0 +1,24 @@
name: Test Issue Gate
on:
pull_request:
paths:
- '.github/scripts/issue-gate.mjs'
- '.github/scripts/issue-gate.test.mjs'
- '.github/workflows/issue-gate.yml'
- '.github/workflows/issue-gate-tests.yml'
permissions:
contents: read
jobs:
test-issue-gate:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
- name: Run issue gate tests
run: node --test .github/scripts/issue-gate.test.mjs

46
.github/workflows/issue-gate.yml vendored Normal file
View File

@@ -0,0 +1,46 @@
name: Issue Gate
on:
pull_request_target:
branches: [main]
types: [opened, edited, reopened, synchronize, labeled, unlabeled]
workflow_dispatch:
inputs:
pr_number:
description: Pull request number to evaluate
required: true
type: number
permissions:
contents: read
issues: write
pull-requests: write
statuses: write
concurrency:
group: issue-gate-${{ inputs.pr_number || github.event.pull_request.number }}
cancel-in-progress: true
jobs:
issue-gate:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
# pull_request_target is privileged. Explicitly check out only the trusted
# default branch and never fetch or execute pull request code.
- name: Check out trusted gate implementation
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
ref: ${{ github.event.repository.default_branch }}
persist-credentials: false
- name: Evaluate ready issue
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Until a cutoff is configured, every PR is treated as legacy. This
# keeps the newly merged workflow inert by default.
ISSUE_GATE_CUTOFF: ${{ vars.ISSUE_GATE_CUTOFF || '9999-12-31T00:00:00Z' }}
ISSUE_GATE_MODE: ${{ vars.ISSUE_GATE_MODE || 'observe' }}
ISSUE_GATE_READY_LABEL: ${{ vars.ISSUE_GATE_READY_LABEL || 'state:ready' }}
PR_NUMBER: ${{ inputs.pr_number || github.event.pull_request.number }}
run: node .github/scripts/issue-gate.mjs