Secrets in Jira Issues: What the Research Found

The stack trace looked more suspicious than the token

A production issue contains a 200-line stack trace, several commit hashes, signed-looking URLs, dummy credentials from a test fixture, and one real bearer token. A broad pattern matcher can flag half the page. A narrow matcher can miss the token. That is the central detection problem inside issue reports: the content mixes natural language, code, identifiers, and operational output in one document.

A recent preprint, Secret Leak Detection in Software Issue Reports using LLMs: A Comprehensive Evaluation, studies that surface directly. The researchers curated 25,000 issue-report instances, of which only 437 were labeled true positives. That means the benchmark is intentionally—and realistically—dominated by safe material that can look secret-like.

Their reported regular-expression baseline achieved an F1 score of 0.0341; their combined approach reported 0.6347. F1 balances precision and recall into one measure. The paper specifically identifies logs, URLs, commit IDs, stack traces, and dummy passwords as sources of noise.

What the numbers do—and do not—establish

The study establishes that issue reports deserve their own benchmark. A detector evaluated only on source files has not demonstrated performance on prose mixed with debugging artifacts.

It does not establish that 1.7% of all real-world Jira issues contain secrets. The corpus was constructed for evaluation, not sampled to estimate population prevalence. It also does not prove that every regex scanner will achieve the reported baseline or that every language model will achieve the reported higher score. Results belong to the dataset, preprocessing, model, baseline, and labeling choices described by the authors.

The work is currently an arXiv preprint. It is useful primary research with a stated method, but it should not be presented as a finished peer-reviewed consensus.

Why Jira makes the classification problem harder

Issue content carries signals that rarely occur together in normal source files:

• a stack trace may contain high-entropy request IDs and a real connection string;
• a runbook may deliberately show a safe placeholder beside an accidentally live token;
• a URL may include an ephemeral signature or a harmless commit hash;
• a copied environment dump may contain both public configuration and reusable authority.

The operational response also differs. A scanner should redact a supported credential quickly, but creating a critical incident for every identifier produces a queue people stop trusting. The detection decision and the escalation decision should therefore be separate.

A practical evaluation for Jira and Confluence

Build a synthetic corpus without live credentials. Include unmistakably fake provider-shaped examples, safe placeholders, hashes, URLs, stack traces, private-key-shaped multiline text, and near-misses. Place them in descriptions, comments, and pages. Record true positives, false positives, and false negatives by content type—not only an overall percentage.

Secret Sentinel is designed for this specific surface. It scans Jira work items and comments plus Confluence pages and comments, redacts detected values, and separates risk-based severity from Jira escalation. Known-safe exact values can be ignored, while Advanced adds custom patterns for internal credential formats. It does not use or claim the paper’s model; the research explains the content problem, not the implementation of this product.

The practical conclusion is narrower and stronger than “AI beats regex”: repository coverage cannot protect an issue it never reads, and issue-report detection must be evaluated against the noise issue reports actually contain.