Secret Scanner Accuracy: Precision Is Not Enough

The quiet scanner that missed the production key

Two tools scan the same corpus. Tool A raises ten alerts and nine are real. Tool B raises 100 alerts and finds 80 real secrets. Tool A has the cleaner queue; Tool B may be protecting far more of the environment. “Few false positives” is not an accuracy claim until we know what the tool failed to report.

A peer-reviewed 2023 ESEM study, A Comparative Study of Software Secrets Reporting by Secret Detection Tools, evaluated five open-source and four proprietary scanners against a benchmark dataset. No product dominated both axes. GitHub Secret Scanner led measured precision at 75%, while Gitleaks led recall at 88%. Gitleaks’ measured precision was 46%; TruffleHog’s recall was 52%. These are results for the versions, configuration, and corpus studied—not timeless product rankings.

Why detectors fail in different directions

The researchers manually analyzed errors. False positives came from generic regular expressions and ineffective entropy calculations. False negatives came from faulty regexes, skipped file types, and incomplete rulesets. That distinction matters operationally:

• A narrow prefix rule can be precise yet blind to internal tokens and passwords.
• A generic token= rule can find unknown formats while matching documentation examples.
• Entropy can identify random-looking strings but also hashes, IDs, and test fixtures.
• Provider validation can confirm some live tokens, but testing a credential against an external API changes the data-flow and threat model.

Build a corpus from your actual failure modes

Do not test with live credentials. Construct synthetic examples that preserve the formats your organization uses: paired identifiers and secrets, multiline private keys, connection strings, tokens in prose, safe placeholders, hashes, and deliberately malformed near-matches. Include Jira comments and Confluence storage-format content if those are in scope; a benchmark made only of source files cannot establish recall in collaboration tools.

Record four counts for each detector: true positive, false positive, true negative, and false negative. Then split results by credential class and content surface. An average can hide a scanner that catches every AWS key and none of your internal service tokens.

Tune response by exploitability

Not every match deserves the same workflow. A paired, usable cloud credential should trigger immediate containment. A bare identifier may justify redaction without paging an incident commander. A known documentation placeholder should be ignored by exact value, not by disabling the detector category.

Secret Sentinel follows that separation in Jira and Confluence: it classifies findings by actual exploitability, supports exact-value ignores and per-type controls, and its Advanced edition adds organization-specific regex rules. That does not make the academic trade-off disappear; it gives administrators levers to manage it without turning off redaction broadly.

The trustworthy procurement question is therefore not “what is your false-positive rate?” It is: “show us precision and recall, by credential type, on content shaped like ours—and show us what happens to each class of finding.”