From Entropy to LLMs: What the Research Says About Detecting Secrets
A 1948 formula is still doing security work
Most secret scanners in production today lean on an idea from before computers had operating
systems. In 1948, Claude Shannon published “A Mathematical Theory of
Communication” in the Bell System Technical
Journal, defining entropy as H = −Σ p(x) log p(x) — a measure of how unpredictable a sequence
is. A real API key looks close to random by design, so it scores high entropy. Scanners exploit
exactly that: flag strings whose entropy crosses a threshold, then narrow with format-aware
regex for known prefixes (AKIA, ghp_, sk-, and similar).
The trouble is that plenty of harmless strings are also close to random: hashes, UUIDs, base64-encoded images, minified JavaScript, and test fixtures. Entropy tells you “unpredictable string,” not “secret.” Truffle Security’s own writeup on how TruffleHog verifies secrets is candid about the consequence: traditional entropy-plus-regex detection produced, in their words, “an incredibly high false-positive to true-positive ratio” — which is why they added a further step, live credential verification against the issuing API, on top of the pattern match.
You can’t measure any of this without a labeled corpus
“Our scanner is 95% accurate” is not a checkable claim without a public, labeled dataset behind it. Two real ones now exist. SecretBench (Basak, Neil, Reaves, and Williams, MSR 2023) manually labeled candidates from 818 GitHub repositories across 49 languages, confirming 15,084 real secrets out of 97,479 candidates — a ground truth researchers and vendors can both test against. CredData, maintained by Samsung, takes a similar approach at larger scale: roughly 19.4 million lines of code, benchmarked against eight open-source scanners including Gitleaks and TruffleHog.
The newest layer: classifying candidates with a language model
A 2025 study, “Secret Breach Detection in Source Code with Large Language Models” (Rahman, Ahmed, Wahab, Sohan, and Shahriyar, ESEM 2025), tested a hybrid pipeline: regex extracts candidate strings first, then a fine-tuned LLaMA-3.1 8B model classifies each candidate as a real secret or not, evaluated on data derived from SecretBench’s source repositories. The fine-tuned model reached an F1 score of 0.9852, ahead of the regex-only baselines tested alongside it. That’s a genuine result — and also a result on one specific benchmark, one specific model, one specific corpus. It demonstrates the technique works, not that any shipped product using it will perform identically on your codebase’s mix of internal tokens, connection strings, and vendor key formats.
What this means for evaluating a scanner
Three real generations of technique now exist — keyword/regex, entropy, and learned classification — and each one only means what it claims to mean in the context of a named, inspectable benchmark. That’s the same conclusion this site’s precision/recall research reached from a different angle: ask a vendor for the corpus and credential-type breakdown behind any accuracy number, not just the number.
Secret Sentinel makes a deliberate trade-off in that landscape: its detection methodology is regex- and rule-based — built on the open-source secretlint engine plus documented additions, running through RE2 for linear-time matching — rather than a black-box ML classifier. That gives up some of the accuracy ceiling a fine-tuned LLM can reach on a benchmark, in exchange for rules an administrator can read, test with the product’s own synthetic examples, and reason about directly. Neither choice is free; know which one you’re buying.
Detection is only half the picture, too. A scanner can classify a string perfectly and still leave the underlying credential valid — pattern matching finds exposure, it doesn’t revoke anything. Reducing how much ever reaches a scanner in the first place is a separate, upstream control: envseal keeps secrets encrypted per-key in git rather than sitting in plaintext for any scanner, human, or attacker to find unencrypted at all.
Frequently asked questions
What is Shannon entropy, and why do secret scanners use it?
Entropy, formalized by Claude Shannon in 1948, measures how unpredictable a sequence is. A real API key or private key looks close to random, so it scores high entropy. The problem is that so do hashes, UUIDs, base64-encoded images, and minified code — entropy alone can't tell "random-looking" apart from "actually secret," which is why it's almost always paired with format-aware regex rather than used on its own.
Are LLM-based secret scanners better than regex and entropy?
On a 2025 academic benchmark, a fine-tuned LLaMA-3.1 8B model reached an F1 score of 0.9852 on binary secret classification, notably ahead of regex-only baselines evaluated on the same data. That's a real, measured result on one benchmark and corpus — it's not evidence that any particular product's ML scanner outperforms any particular product's regex scanner in your environment, which still has to be measured directly.
Should I trust a vendor's advertised detection accuracy number?
Only if it names the benchmark, the corpus, and what counts as a true positive. Published academic results (SecretBench, CredData) use specific, documented datasets — a number with no stated corpus, credential mix, or content surface behind it isn't independently checkable.