Guinea Pig: a reference test bed for benchmarking security scanners

Every scanner vendor claims to catch everything. I wanted a way to check: honest, reproducible, and where the ground truth isn’t up for debate. So I built Guinea Pig: one CWE per directory, every vulnerable fixture paired with its remediation, a synthetically-generated catalogue of 1,028 secrets, and a benchmark that scores recall, false positives and noise. Here’s how it’s built, and what a late-June run actually said.

Introduction🔗

If you run static scanners, you already know the uncomfortable truth: nobody can tell you how good they are. Every vendor’s page says “detects OWASP Top 10, 700+ rules, industry-leading precision”. None of them run on the same corpus, so the numbers don’t compose. And the moment you try to compare two SAST tools yourself, you hit the real problem, which isn’t running them. It’s knowing what the right answer was. Did the scanner miss a bug, or was there no bug there? Did it flag a false positive, or did you mislabel the file?

So I built a small, boring, reproducible answer to that: a test bed where the right answer is unambiguous by construction. The guiding principle runs through the whole thing:

The path is the ground truth. A fixture at static/CWE-89/python/vulnerable.py is, by definition, a CWE-89 positive. Its patched.py sibling is, by definition, a must-not-flag negative. Nothing is hand-labeled; the layout is the label.

That one decision is what makes the benchmark honest: the scoring is derived from the tree, not maintained in a spreadsheet that drifts. This post walks through the design, the two sub-projects that make it interesting (a synthetic secret catalogue and a context-stripped set for testing agents), and the trends from an actual run.

The scanner engine itself is vendored from my earlier Deep Scan experiment, so the tools scored here are the same 20+ scanners that project wraps. Guinea Pig is the corpus; Deep Scan is the engine. This post is about the corpus.

1. The design: one CWE per directory🔗

Fixtures come in pairs🔗

The atom of the whole repo is a pair of files:

static/CWE-<n>/<language>/vulnerable.<ext>   # a positive: expect a finding of CWE-<n>
static/CWE-<n>/<language>/patched.<ext>      # a negative: the fix, expect NO finding

The <language> folder does double duty: it’s the language and it selects the scanner family that’s supposed to fire. java, c, cpp, csharp, python, javascript are SAST; terraform, ansible, kubernetes are IaC; github, gitlab are PIPELINE; env, ini, conf, pem are SECRET. A three-line header at the top of each fixture (in that language’s comment style) carries the CWE name, a description, and the vulnerable sink, enough for the generator to build a full ground-truth record without anyone hand-writing JSON.

Why the pairing is the whole point🔗

Anyone can collect vulnerable code. The half that people skip is the fix next to it. That patched sibling is the single most valuable file in the repo, because it’s the precision test: a scanner that flags patched.py is producing a false positive on remediated code, and that’s exactly the failure mode that makes teams turn scanners off. Recall alone is vanity. A scanner that flags everything has 100 % recall and is useless. The paired negative is what keeps the benchmark honest about that.

The ground truth is generated, never written🔗

There’s a golden rule in the repo, and I mean it literally:

Note. The benchmark/ground-truth/*.json files are generated. Never edit them by hand. The generator overwrites them. The fixture (its path, plus a small sidecar for the handful of things a path can’t encode) is the source of truth.

Adding a vulnerability is: drop vulnerable.<ext> and its patched.<ext>, write the 3-line header, run the generator:

# derive ground truth from the fixture tree; prints the counts it found
python3 benchmark/src/build_report.py

The corpus today, one CWE per directory, spans a broad matrix:

2. Secrets at scale: a synthetic catalogue🔗

The problem with hand-writing secret fixtures🔗

Secret detection is where a hand-built corpus quietly lies to you. You’ll write a test for the AWS key, the GitHub token, the Slack webhook, the dozen providers you already think about. You will not write fixtures for the 500 providers you’ve never heard of. So your corpus is biased toward exactly the secrets your tools already catch, and it flatters every scanner equally. The blind spots stay invisible because you share them.

secret-rules: union the tools’ own rulesets, then regenerate🔗

The fix was to stop inventing secrets and instead harvest them from the scanners themselves. That’s a separate project, secret-rules: it ingests the native rulesets of five secret scanners, canonicalizes and deduplicates every regex into one referential, and then regenerates config for seven scanners and synthesizes example secrets from the rules.

rules/ (5 source rulesets)          referential/ (canonical)          configs/ + examples
  betterleaks-v1.4.1   ─┐             rules.yaml   (every regex,   ┌─>  gitleaks / betterleaks
  gitleaks-v8.30.1     ─┤   parse +   deduplicated, provenance,    ├─>  kingfisher / noseyparker
  kingfisher-v1.102.0  ─┼─> dedup ──> keywords, entropy, examples) ├─>  titus / trufflehog / YARA
  noseyparker-v0.24.0  ─┤             topics.yaml  (indexed by     └─>  per-rule secret EXAMPLES
  titus-v1.2.2         ─┘             provider/topic)                   → guinea-pig catalogue

The dedup is the interesting engineering: source patterns target different regex engines (RE2 for gitleaks/betterleaks, the Rust regex crate for the others), so before two rules can be compared they’re canonicalized: verbose (?x) patterns flattened, comment groups stripped, named groups made plain, inline flags rewritten to a portable scoped form. Two rules with the same canonical pattern merge: provenance accumulates, keyword and example lists are unioned, and the lowest entropy threshold wins (favouring recall). Each rule carries examples:, positive samples that match its canonical pattern.

From referential to test corpus🔗

Those per-rule examples are the catalogue. Guinea Pig ships them as static/CWE-798/catalogue/<provider>/*.txt: 1,028 per-rule secret positives across 587 providers, plus 21 hand-written benign look-alikes under static/CWE-798/false-positives/ for precision (documented example keys, placeholder .env files, JWT public demos, bcrypt hashes, i18n strings…).

There’s a subtlety worth stating plainly, because it shapes how you read the results in §5:

Watch out. The catalogue is the union of every scanner’s ruleset. By construction, no single scanner can score 100 % against it: each one only knows its own patterns. That’s not a bug in the corpus; it’s the entire point. The union is a mirror that shows each tool exactly which providers it can’t see.

3. The LLM twins: hiding the answer from the agent🔗

Everything that makes the corpus fair to SAST makes it unfair to an agent🔗

Here’s a wrinkle I didn’t see coming. The whole design of Guinea Pig (the CWE in the path, the descriptive filename, the 3-line header naming the sink) exists to make the ground truth obvious. That’s fine for a regex or AST scanner: opengrep doesn’t read the folder name or the comment; it matches code.

But the moment you point an LLM or an agent at static/CWE-89/python/vulnerable.py that opens with # CWE-89: SQL Injection, you’re not testing detection anymore. You’re testing reading comprehension. The model doesn’t have to find the bug; you handed it the answer key in the path and the first line.

Context-stripped, UUID-renamed twins🔗

So I had to build a second tree that deliberately throws away everything the rest of the repo works so hard to encode:

llm/<language>/<uuid>.<ext>     # same code, no telling filename, no CWE header, no guiding comment

98 files, each a twin of a real fixture: identical vulnerable code, but the filename is an opaque UUID, the CWE header is gone, and any comment that would nudge the reader is stripped. Nothing in the tree tells you what to look for.

The map from uuid → original fixture lives out of band, in benchmark/src/llm_sources.json, precisely so it can’t leak from the file tree. The generator reads that sidecar and copies the source fixture’s weaknesses onto the twin, so the stripped file is still scored against the same ground truth as its named original. You get an apples-to-apples comparison: how well does an agent do when it can’t cheat off the labels?

Note. This was the fiddly, slightly ironic part of the build. I spent most of the repo making the answer derivable from the path, then had to build a parallel tree whose only job is to make the answer un-derivable. Testing an agent honestly means removing every affordance you’d normally add for a human reviewer: the helpful name, the leading comment, the tidy folder. If your eval corpus names the file sql_injection.py, you are measuring the label, not the model.

4. Scoring: recall, false positives, noise🔗

The benchmark runs the applicable scanners per theme, maps each finding back to the ground truth, and reports a recap table. The columns are the whole story:

The (✓cwe) distinction matters more than it looks: a scanner can find a secret without labelling it CWE-798, so Found can be high while the CWE-accurate count is zero. Detection and classification are different skills, and the table keeps them separate.

Running it is three commands (the scanners are vendored in-process):

python3 benchmark/src/build_report.py                    # refresh ground truth
uv run python benchmark/src/benchmark.py install --all   # install the scanner tools
uv run python benchmark/src/benchmark.py run             # run + score all themes

Output lands as benchmark.md (recap + top-3-per-family tables), benchmark.json (machine-readable), and HTML/PDF for the humans.

5. What a late-June run said🔗

I won’t dump the full scorecard here: trends are more honest than a leaderboard, and a single run is a snapshot, not a verdict. But a late-June 2026 run over the static theme lined up cleanly with the design, and three trends are worth calling out.

Secrets: nobody clears ~63 % on the union catalogue🔗

Exactly as predicted in §2. Against the ~1,000-entry union catalogue, the field splits into a leading cluster and a long tail:

No tool broke ~63 %. That’s the union effect made visible: each scanner detects roughly the slice of providers its own ruleset covers, and the catalogue is the sum of all of them. Read the right way, this isn’t “titus wins”, it’s “even the best single scanner is blind to a third of the providers some other scanner already knows about.” The argument for a merged referential writes itself.

Maybe my next test will be to test all scanners with the same set of consolidated rules to compare from a same referential. But at the end scanners embed the best tested set of rules for it. An interesting testing bed to consider is that CredData test bed full of real examples.

Precision: the look-alikes earn their keep🔗

The 21 benign look-alikes did their job: documented AWS example keys, Stripe test keys from the docs, placeholder .env files, JWT public demos and bcrypt hashes reliably tripped several secret scanners into false positives. Counts stayed low (single digits per scanner), so precision is good, but not perfect, and the corpus pins down exactly which confusable inputs each tool mishandles. That’s the regression set you actually want: the near-misses, not the obvious keys.

Code & IaC: the patched files bite back🔗

For SAST, recall sat in the ~40-50 % band (opengrep and the code-graph analysis engine led, codeql behind), and the paired patched.* negatives caught real over-reporting: every SAST tool flagged at least a handful of already-fixed files. IaC recall was higher (checkov strongest, ~88 %), but it, too, flagged patched Terraform and Kubernetes. And a meaningful slice of positives (notably C/C++ memory-safety bugs like use-after-free, double-free, and format string, plus a few scanner-specific-only secrets) went undetected by every configured scanner, which is the corpus doing its most useful job: showing you the gaps nobody covers.

Caveat. These are synthetic, regex-derived secrets and small single-sink fixtures. They favour pattern-matchers and won’t tell you how a tool does on a sprawling real codebase. Treat the numbers as relative signal on a controlled subject (a guinea pig), not an absolute grade.

Summary🔗

Guinea Pig is on GitHub; the secret referential that feeds it is secret-rules.

The four principles I’d carry to any scanner evaluation: make the path the ground truth so scoring is derived, not maintained; pair every positive with its fix so you measure precision, not just recall; build your secret corpus from the tools’ own rules so it exposes blind spots instead of sharing them; and strip the answer from the tree before you let an agent near it, or you’re grading the label.

Guinea Pig is a controlled subject, not a verdict machine: synthetic fixtures, single sinks, one snapshot per run. Its value is that the ground truth can’t be argued with, which is exactly what you need before you trust any number a scanner prints at you.