Acceptance Criteria for AI Agents: 3,375 Criteria Checked
On this page
“Review form.” That is a whole acceptance criterion, copied from a public GitHub issue this month. A person on the team knows which form, what “review” means and when it is done. An AI coding agent will build a form, call it reviewed, and write a test that proves its own guess.
Acceptance criteria are suddenly everywhere. GitHub’s search counts about 41,000 new issues a day that mention them, 22 times the rate of a year before. Many are written for agents, some by agents. We checked 3,375 of them to see how many an agent could actually prove.
Short answer: an AI agent can only prove a criterion that names something a test can observe: a result, a number, a state, a “must”. In 3,375 criteria from 969 public GitHub issues, 39.6% had none of that, 0.9% used Given/When/Then and 16.7% had a number. Write each criterion as a check: “Given …, when …, then the API returns 400 with reason refund_window_expired”. Keep to about eight per work item, say what is out of scope, and ask the agent to finish with one line per criterion: the test that proves it, or “not proven”.
Acceptance criteria are growing faster than ever
GitHub’s issue search counts public issues whose text contains “acceptance criteria”. Per day, by the quarter they were created: about 1,900 a day in July to September 2025, 15,100 in April to June 2026, 41,200 from 1 October to 17 November 2026. The counts are GitHub’s own estimates and include issues created by tools; the shape is what matters. Since coding agents started taking issues, teams write criteria for them, and agents write criteria for each other.
What 3,375 criteria look like
How we sampled. On 18 November 2026 we took up to 200 issues from GitHub’s search for each of six periods since July 2025, 1,199 in all, found the acceptance criteria section (a heading, a bold line or “Acceptance criteria:”), and read its list items: 969 issues had a list, 3,375 criteria. We checked each with Agent Ready’s own wording rules: a vague-word list (“fast”, “user-friendly”, “etc.”, “appropriate” and 18 more) and a lenient test for “something an agent can check” (a verb of an observable result such as shows, returns, saves or rejects; a state such as is or stays; a number; “must”; or Given/When/Then).
| What we measured | Result |
|---|---|
| Criteria per issue, median (90th percentile) | 3 (6) |
| Words per criterion, median | 8 |
Issues with checkboxes - [ ] | 67.1% |
| Criteria with no observable result, number or must | 39.6% |
| Issues with at least one such criterion | 64.2% |
| Criteria with a number | 16.7% |
| Criteria in Given/When/Then form | 0.9% |
| Criteria with a vague word | 1.2% |
| Issues with more than 8 criteria | 2.6% (25) |
| Placeholder criteria (“TBD”, “n/a”, ”…”) | 36, in 31 issues |
Three things stand out. Criteria are short: eight words, three per issue. They are rarely vague in the classic sense: “fast” and “user-friendly” are almost gone. And the real problem is noun phrases: “Review form”, “Leader election pattern explained”, “Clickable tracking link”, “Required resources/approvals listed”. Each names a thing, not a result. A person fills the gap from context; an agent fills it with a guess.
Our pattern is a heuristic and it errs both ways: “Builds successfully” counts as untestable to it because “builds” is not on its verb list, and some criteria were in German or Indonesian. Read 39.6% as “a lot”, not as a precise figure.
From a label to something an agent can prove
Real criteria from the sample, and a version a test can check:
| From a public issue | Provable version |
|---|---|
| Review form | Given a submitted expense, when a manager opens it, then the form shows the amount, the receipt and Approve and Reject buttons |
| Leader election pattern explained | The README section “Leader election” has a sequence diagram and covers what happens when the leader stops |
| Clickable tracking link | The order e-mail contains a link that opens the carrier’s tracking page for that parcel |
| Required resources/approvals listed | The ticket lists every approver by role; the workflow blocks Start until all of them approved |
| Builds successfully | make check-all passes on the pull request |
The pattern: name who or what starts it, what happens, and what you can see at the end, with the number if there is one. If you cannot write the last part, the work is not ready, for a person or an agent.
Rules for criteria an agent can prove
- One result per line. “Saves the draft and e-mails the reviewer” is two criteria, and an agent can prove one and claim both.
- Observable, not internal. “Uses the cache” is hard to test from outside; “the second request returns in under 100 ms” is not.
- Numbers everywhere they exist. Limits, rounding, retries, time zones, counts. 16.7% of public criteria had one.
- Boundaries. Day 30 and day 31; zero items and one; the empty string.
- Out of scope, written down. It stops the agent from “helpfully” doing the next ticket.
- About eight at most. More is several agent runs. Agent Ready’s Strict for agents preset flags a work item with more than 8 criteria or 800 words and drafts one sub-task per criterion.
- No placeholders. “TBD” under the heading is not a criterion; Agent Ready does not count it.
Given/When/Then is a good default when the situation matters. More examples for people, and the Jira side: acceptance criteria: 18 examples. The full readiness list, one story rewritten four times: Definition of Ready for agents.
Make the agent show the proof
A pull request that “meets the acceptance criteria” is a claim. Ask for the evidence instead, one line per criterion:
AC1 Given a payment older than 30 days ... returns 400
proven by: refunds.test.ts › "rejects a refund on day 31"
AC2 Payments of 30 days or less refund as today
proven by: refunds.test.ts › "refunds on day 30"
AC3 A test covers day 30 and day 31
not proven: day 31 covered, day 30 only through AC2
Agent Ready makes this the agent’s last step. Through its MCP tools, an agent finishes a work item with
finish-issue and a receipt: for each criterion the test that proves it, or “not proven”, plus what
changed and the risks. The work item shows “2 of 3 criteria proven by tests” with a bar of one segment
per criterion. The receipt is then checked against the pull request and builds Jira knows about:
Verified, Unverified (no pull request yet) or Contradicted (a red build), checked again for
14 days. In our Claude Code runs, two of three work items finished with 4 of 4
criteria proven; the third stopped honestly because its criteria were placeholders.
Check criteria before the agent starts
The cheapest fix is before the handoff. Agent Ready checks every Jira work item for an acceptance criteria section with something under it, and, with Clear wording on, for vague words and criteria with nothing to test, using the same rules as this study. Write a criterion step by step turns three short answers into one “Given …, when …, then …” line, with no AI. On sites with Rovo, its agent drafts missing criteria from the ticket’s own text and marks what it does not know as questions. The Issue is agent-ready validator keeps a work item without criteria out of Ready for agent.
Sources
- GitHub: issue search counts for “acceptance criteria” by creation date, read 18 November 2026 (GitHub’s estimates).
- Our sample: 1,199 public GitHub issues from the same search, 969 with a list of criteria, 3,375 criteria, checked with Agent Ready’s wording rules (vague words and its testable-criterion pattern) on 18 November 2026.
- Related: Rovo agents in Jira, tested (2 of 7 runs invented facts), Definition of Ready.
Frequently asked questions
How do you write acceptance criteria for an AI coding agent?
Write each one as something a test can check: a situation, an action and a result you can observe, such as "Given a payment older than 30 days, when a refund is requested, then the API returns 400 with reason refund_window_expired". Add the numbers (limits, counts, formats), keep to about eight criteria per work item, and say what is out of scope.
Can AI write acceptance criteria?
It can draft them from what the ticket already says, and you must check every number and name. In our test of Rovo agents on Jira, 2 of 7 runs invented facts when asked to fill criteria the ticket had no facts for. Let an agent draft, then a person who knows the work decides.
Do acceptance criteria have to be Given/When/Then?
No. A rule ("Totals round to two decimals") is fine when it names an observable result. Given/When/Then helps most when the situation matters. In our sample only 0.9% of criteria used it; far more common was a noun phrase like "Review form", which a test cannot check.
How do I know an agent met the acceptance criteria?
Ask it to prove each one: for every criterion, the test that checks it, or "not proven". Then check the claim against the pull request and its build. Agent Ready makes this the agent's last step through MCP, a receipt with one line per criterion, marked Verified or Contradicted against Jira's development information.
Keep reading
- Jira, JSM and the Marketplace · 13 minDefinition of Ready: Examples and the AI Agent Version
- Jira, JSM and the Marketplace · 9 minAcceptance Criteria: 18 Examples, Given/When/Then and Jira
- Jira, JSM and the Marketplace · 6 minSkills vs MCP vs AGENTS.md: Which to Use, and When
- Free toolSKILL.md validator