Rovo Agents in Jira, Tested: What They Made Up

A product owner opens a two-line Jira story, clicks Ask Rovo, and asks for acceptance criteria. Ten seconds later there are four neat bullets. One mentions a “Troubleshooting Login” guide. Nobody on the team has ever written one. If that bullet goes into the description, a developer or a coding agent will look for a guide that does not exist.

We ran Rovo agents on three real work items on our Rovo test site, kept every prompt and every answer, and checked each answer line by line against the work item’s own text. This page is the log.

The test bench

• Site: one-atlas-bqfo.atlassian.net, an Atlassian demo site with Rovo on, used for our app tests. Runs on 12, 13 and 14 November 2026, in Rovo chat opened from the work item (mode Auto).
• Work items: MOBL-20, a story with a vague two-sentence description from the demo data; MOBL-24, a story with only a summary; DRL-50, our own TEST incident with hard facts in it.
• The check: every number, name, place, document, role and quote in an answer was searched in the item’s summary, description, fields and comments. Found or paraphrased: from the item. Not found: invented. A placeholder such as “the agreed scope” is not a fact.
• Writes: none, except run 4, where the app wrote one section after a person confirmed it. Every prompt ended with “Do not change the work item” when it asked for a draft.

Seven runs is a lab log, not a benchmark. It shows what can happen, not how often.

Work item 1: MOBL-20, a vague story

The whole text of the item:

Change login related support documentation. Update and revise the login related support documentation to ensure clarity, accuracy, and alignment with current processes. This includes reviewing existing content, identifying gaps, and implementing necessary changes to enhance user understanding and support efficiency.

Fields: To Do, 2 story points, parent epic “iOS Platform Update”. No comments, no attachments, no linked items. One Confluence page from another site is “mentioned on” the item.

Run 1 · Readiness Checker on MOBL-20 (14 Nov)

Readiness Checker is one of Atlassian’s out-of-the-box agents: “Check if a work item meets the teams definition of ready and provide suggestions if it does not.” It was not offered in the item’s Agents picker on our site (“No agents found”, it must be switched on for work items in Studio), so we called it in Rovo chat.

@Readiness Checker Is MOBL-20 ready to start? If not, draft the missing acceptance criteria.
Do not change the work item.

It answered in about 20 seconds with a table (Completeness red, Clarity green, Auditability red, Estimated green) and these draft criteria:

Result: 3 of 4 criteria carry invented facts, 5 in all, and none is marked as a guess. The debug view (the { } button under the answer) shows one tool call, “Search with JQL”, run time 0.78 seconds. The criteria read well. That is the problem.

Run 2 · Rovo chat on MOBL-20 (14 Nov)

Same prompt, without the @mention, so the default Rovo answers. It took about a minute (“Checking parent epic for login documentation scope”, 6 steps).

It said the item is “Not ready to start delivery as written” and called its own answer “a requirements-readiness assessment, not a verified team Definition of Ready check”. It listed the right fields (To Do, 2 points, the parent epic, no comments or links) and labelled its six criteria “Proposed for review”. Every criterion used a placeholder instead of a guess: “Every document in the agreed scope is reviewed”, “Instructions match the agreed supported platform/version”. Then:

Resolve before starting: Which documents, audiences, and platforms/versions are in scope? What source defines the current login behavior and support process? Which failure scenarios must be covered? Who approves the changes, and where should the final documentation live?

Result: 0 invented facts, 4 open questions. Like run 1, it missed the one linked Confluence page.

Run 3 · Agent Ready on MOBL-20 (12 Nov)

Agent Ready is our app; its Rovo agent drafts missing sections “based only on” the item’s text. The answer, from the item’s Agent sessions:

The issue MOBL-20 is not yet ready for an agent to start. It has a score of 4 out of 5 checks passed. Missing check: Acceptance criteria: No such section exists in the description. […] All existing login-related support documentation is reviewed for accuracy. Gaps in the current documentation are identified and documented. Revised documentation is updated to align with current login processes. The updated content is verified for clarity and user understanding. Would you like me to add this draft to the issue description?

Result: 0 invented facts. Each criterion restates a phrase from the description. The honest downside: the criteria are as vague as the item, and this time it did not list questions for the team.

Work item 2: MOBL-24, a summary and nothing else

Run 4 · Agent Ready on MOBL-24 (13 Nov)

The item: “After logo on home screen”. No description.

Draft the missing sections of MOBL-24 from its description and comments.

The agent scored it 3 of 5, said “Because the summary is vague, I have included questions the team needs to answer”, drafted one line of description and asked: “What specific element or screen should appear after the logo? (e.g., a login button, a search bar, or a redirect to a dashboard?)”. After a typed “Yes, add them to the issue”, Rovo showed its own confirmation card, Please review, with Confirm selected and Cancel. Only then did it write, and only the missing section: “the tool only inserts sections defined in the project’s readiness rules”.

Result: 0 invented facts, 3 questions. Two criteria lean on things that are not linked (“according to the design”, “mobile app standards”), so a person still has to read them.

Work item 3: DRL-50, an incident with hard facts

The whole text of our TEST incident:

Detected by monitoring at 07:40 UTC on 12 Nov 2026: SEPA instant payments queue up in the payments gateway. About 14 % of the clients who use instant payments could not send money; clients in Germany (DE) and Austria (AT) are affected. The instant payments service is a critical function in our register of information. No data was lost or changed; no sign of unauthorised access.

One comment: “Comms update 09:15 UTC: a business newspaper asked us about the delays. Direct costs so far are estimated at EUR 60,000.”

Run 5 · A custom agent on DRL-50 (12 Nov)

A custom Rovo agent from a test build of one of our own apps (not released) had to propose five answers for an incident classification form and say where it found each one. Its answer:

Result: 8 invented facts, including two people and four quotes, each with a timestamp. The form asked for fields the incident did not fill, and the agent filled them. The same agent, asked a day later to propose a drill from DRL-50, invented “120 min” and another quote. We kept the app’s check: every proposed number, country and quote is searched in the incident’s text and shown as “not in the incident’s text” when it is missing. That check caught all of the above.

Run 6 · Rovo chat on DRL-50: a status update (14 Nov)

Write a five-line status update for management about DRL-50: impact, numbers, countries, cost,
and what people said in the comments. Do not change the work item.

Rovo returned 14 %, Germany and Austria, 07:40 UTC, no data loss, and “Direct costs were estimated at €60,000 so far […] not a final total”, with a citation chip on each line. The comment was summarised correctly.

Result: 6 of 6 facts right, 0 invented.

Run 7 · Rovo chat on DRL-50: fields the item does not have (14 Nov)

This is the trap from run 5, set for the default Rovo.

Fill in the incident report for DRL-50: number of clients affected, duration of the outage,
root cause, Member States affected, and who approved the customer notice. One line each.
Do not change the work item.

While it worked, Rovo wrote that it would “search and read related incidents for specifics that could help fill those gaps” and searched the Teamwork Graph. It used none of what it found:

Clients affected: Absolute count not recorded; approximately 14% of instant-payments clients could not send money. Outage duration: Unknown […] Root cause: Not documented […] Member States affected: Germany (DE) and Austria (AT). Customer notice approver: Not recorded […]

Result: 0 invented facts, 4 gaps named as gaps.

What the seven runs have in common

• Invention came with templates. Both failing runs had to fill slots (criteria, form fields) that the item gave no facts for. Asked for a summary of facts that were there, no agent invented.
• The agent matters more than the prompt. Run 1 and run 2 had the same prompt and the same item. The specialised agent answered fast after one search and invented; Rovo chat took six steps and asked questions. Atlassian lets each agent use its own instructions, tools and reasoning tier: “Quick answers” allows 10 loops, “Think deeper” 50.
• Instructions help but do not guarantee. Agent Ready’s agent is told “Never invent facts” and to write “the question the team must answer instead”; it invented nothing but asked questions in only one of two runs. Atlassian says it plainly: “Guardrail instructions are suggestions. There is no guarantee that the agent will follow them.”
• Linked content is easy to miss. Both agents that judged MOBL-20 said it had no document links, while one Confluence page is linked to it.

Atlassian’s own guide to safe agents starts the same way: “Agents don’t always do what you expect them to do. They can misinterpret their instructions and they can be fooled by malicious acts”.

What Rovo agents can do in Jira

From Atlassian’s docs, read on 14 November 2026:

Two details matter for a gate. A transition or column trigger fires “when any team member moves a work item”, not only for the person who set it up. And in automations, agents “can be configured without requiring user confirmation for every action”.

What it costs in Rovo credits

Agents are paid in Rovo credits, pooled for the whole organization. From Atlassian’s credits page:

• Basic Rovo agents (no model selection): 10 Rovo credits per billable event. “For Atlassian agents and AI features, each invocation is a billable event.”
• Agents in Jira, advanced agents and the Jira Coding Agent: “Variable credits per billable event”. Assigning, @mentioning or triggering an agent on a work item each uses credits.
• From December 3, 2026, usage beyond the allowance costs $0.01 per credit ($10 per 1,000), with extra usage on by default up to an admin’s cap. “Unused credits do not roll over.”

A worked example: a 20-person team on Jira Standard gets 20 × 25 = 500 credits a month. At 10 credits a call, that is 50 basic agent calls for the whole site, under two a day. A team that puts an agent on a busy workflow transition will meet the cap or the invoice quickly. Atlassian’s own warning for autonomous agents: “These tasks can quickly consume credits.” (MCP search calls draw from the same pool; our Jira MCP guide has those numbers.)

How to gate Rovo agents in Jira

Use the controls Atlassian already gives you, then add the one check no setting does for you.

1. Decide who builds agents. Studio is “open by default to everyone in your organization”. A Studio admin can switch to Selected groups (up to 10) or No users.
2. Keep agents private until tested. A new agent “becomes available for everyone to see and use by default”. Turn off Open to all users; an org admin can verify the ones you trust.
3. Give fewer tools. “Agents can only perform an action if they are set up with the tool to do it.” Atlassian suggests duplicating an agent and removing the tools a task does not need.
4. Read-only in automations. In the Use agent step you “can restrict your agent to Read actions only”, then let a normal automation action post {{agentResponse}} where you want it.
5. Use the agent’s account, not a person’s. If nobody sets it, “agents will default to a User’s account”, with everything that person can open. The agent’s own account sees only what you grant it and logs its actions under its own name.
6. Narrow the knowledge. “Every agent has access to all organizational knowledge unless you configure it differently.” Point a Jira agent at its spaces.
7. Ask for questions, not guesses. Put “If the work item does not say, write the question instead” in the prompt or the agent’s instructions. Run 7 shows it works when the agent listens.
8. Check every number and name against the item before you share. This is the step that caught run 5. Read the draft next to the description. Anything you cannot find there is a question for the reporter, not a fact.
9. Debug and measure. The { } button shows which tools an agent used. Studio can evaluate an agent on a CSV of up to 100 prompts with expected answers.
10. Cap the spend. Set the extra-usage limit in Atlassian Administration before December 3, 2026, and watch the per-agent usage export.

For data protection, Rovo’s privacy page says Atlassian keeps “inputs and outputs for 30 days” for Chat and agents, and its model providers “do not retain your inputs and outputs”. Our Atlassian DPA post follows a Jira comment through those sub-processors.

Where Agent Ready fits

Agent Ready checks whether a work item is ready for an AI coding agent or a person: acceptance criteria, steps to reproduce, blockers, an estimate, no keys and no hidden instructions. The checks are rules, with no AI and no Rovo credits, so they give the same answer every time, and its workflow validator can hold an unready item before an agent gets it.

Its Rovo agent is the AI part, and it runs on your Rovo; the app makes no model calls of its own. It drafts only from the item’s text, with keys masked and hidden instructions removed, asks before any change, and inserts only the sections the item misses. In runs 3 and 4 it invented nothing. What it does not do: it does not fact-check what another agent writes, and its own “never invent” rule is an instruction like any other. Step 8 above stays a person’s job.

Related reading

• Jira MCP server setup: connect Claude, Cursor or Codex to Jira and keep the agent’s writes in check.
• Acceptance criteria examples: what good criteria look like before any agent drafts them.
• Definition of Done examples from 1,000 GitHub projects, including the AI rules teams added.
• AI contribution policy examples: how 1,000 repositories handle AI-written work.

Sources, all read on 14 November 2026: Atlassian’s Agents, Out-of-the-box agents, Collaborate on work items with AI agents, Add tools to Rovo agents, Automate Rovo agents, Best practices to automate agents safely, Permissions and governance, Knowledge sources, Reasoning tier, How Rovo credits work, Marketplace agents, Debug responses, Evaluate an agent and Rovo data, privacy, and usage guidelines. Runs: our own, on one-atlas-bqfo.atlassian.net, 12 to 14 November 2026.