Snyk's Own Benchmark Just Admitted LLM Security Scanners Have a Repeatability Problem

Here’s an uncomfortable truth from a vendor that sells both SAST and AI security tooling: if you run the same LLM security review twice on the same code, you get two different bug lists. According to Snyk’s newly published VulnBench JS 1.0 benchmark, nearly 50% of vulnerability reports that LLMs flag outside a deterministic SAST reference appeared in only one of five identical scans. That’s not a rounding error. That’s a product reliability question that any team routing AI code review into their CI pipeline needs to answer before shipping.

Snyk ran 300 scans across 10 JavaScript fixtures, six configurations, and five repetitions each. The setup was intentionally boring — small Express apps, the same prompt, the same harness. The point wasn’t to crown a winner. It was to measure whether an agentic security reviewer, given identical inputs, produces identical outputs. The answer is: sometimes, and only for certain classes of findings.

Why Repeatability Is the Real Security Metric for AI Code Review

Snyk’s core finding splits neatly in two. When Claude matched a Snyk Code reference vulnerability, it did so reliably: 134 of 158 unique reference-matched findings (84.8%) appeared in all five repetitions. When Claude reported bugs outside the reference set, the picture collapsed. Of 161 unique-unmatched findings across 250 model runs, 80 appeared in only one of five runs. Just 22 appeared in all five.

This matters because coding agents are increasingly the first eyes on a pull request. If your reviewer’s queue looks materially different on Tuesday than it did on Monday for the same diff, triage time explodes and developers stop trusting the tool. Traditional SAST is boring on purpose — same code, same rules, same output. LLM reviewers are trading that determinism for the ability to reason about unfamiliar code, and Snyk’s data suggests the trade isn’t free.

If you’re a platform team piping LLM security findings into GitHub check runs, this means you can’t treat an LLM verdict as a stable signal for blocking merges. A finding that shows up once might be a real bug, a false positive, or just model variance. Our take: within 18 months, any serious AI security review product will publish a per-finding confidence score derived from repeated internal sampling — because “the model said so” is not going to survive contact with production engineering culture.

The Complementarity Argument Snyk Is Quietly Making

Read between the lines and Snyk isn’t pitching LLMs as SAST replacements. It’s pitching them as SAST supplements. The benchmark’s highest-recall LLM configuration found only 81% of Snyk Code reference vulnerabilities. The best-scoring LLM configuration hit 75.4% Snyk-reference F1, leaving a 24.6-point gap versus deterministic SAST reference reproduction. And the highest-recall LLM also carried the noisiest queue, with 41% of its reports falling outside the reference set.

But the interesting data point cuts the other way too. In the js-project-nightowl fixture, all 25 model runs flagged a SQL injection that Snyk Code missed — and Snyk’s own report says that finding “is likely a real product gap to investigate.” Meanwhile, in js-project-tigerteam, models cried SQL injection 25 out of 25 times on a mock dbQuery() helper that just logs a string and returns an empty array. No sink, no exploit, no bug. LLMs mistook vulnerability-shaped code for an actual vulnerability.

This is why teams building supply chain and logistics platforms — where a single missed data-flow sink can compromise auditability — should not treat any single scanner as authoritative. Run deterministic SAST for coverage and enumeration of repeated sinks. Run agentic review for the odd business-logic case that a rule engine can’t express. The prediction: expect Snyk, Semgrep, and GitHub Advanced Security to ship native “SAST + LLM ensemble” modes as first-class products in the next release cycle, with confidence scoring built on exactly this kind of repeatability data.

Why the Newest, Most Expensive Model Wasn’t the Best Reviewer

Here’s the counterintuitive bit. Claude Opus 4.7 Max — the priciest configuration in the benchmark — averaged 95,969 tokens and $0.3559 per session. Claude Opus 4.6 Medium averaged 51,574 tokens and $0.0628. That’s 5.67x more cost and 1.86x more tokens for Opus 4.7 Max, and it scored worse: 68.8% Snyk-reference F1 versus 75.4% for Opus 4.6 Medium. Opus 4.7 Max also produced 36 unique unmatched findings, 47.2% of which appeared in only one of five runs. Opus 4.6 Medium produced 5 unmatched findings, none of which were one-offs.

Why it matters: procurement teams are going to see the newest model number, assume it’s better, and pay 5x more for a noisier queue. The Snyk fixtures are tiny, so the absolute dollar difference is trivial. Scale that to a monorepo running security review on every PR across hundreds of engineers, though, and you’re looking at a real cost line — with worse signal quality.

If you’re an engineering leader budgeting for AI security tooling in 2026, this means the “use the biggest model” default is wrong. Benchmark the specific configuration against your codebase and your finding types before locking in a contract. Our prediction: the AI security tooling market is about to bifurcate into “cheap fast reviewer” and “expensive deep reviewer” tiers, and the cheap tier will win most of the CI budget because determinism and cost matter more than marginal recall.

The Blind Spot Nobody Talks About: App-Like Fixtures

The smaller the fixture, the better the models looked. On the largest app-like fixture, js-project-nightowl — 198 lines in server.js plus another 183 lines across db.js and public/app.js, with routing, uploads, and database state — Claude Opus 4.6 High was perfectly stable but stable at only 40.0% Snyk-reference F1. Across five repetitions, it missed every path-traversal reference finding and two of three resource-limit findings.

The failure mode is telling: the model found some representative issues, then stopped enumerating repeated vulnerable sinks. It scanned like a human — spot check the first attachment flow, assume the pattern holds — and missed the second and third instances of the same bug. Deterministic data-flow analysis doesn’t have that laziness problem. That’s the actual complementarity argument.

For teams building regulated systems — think HIPAA-compliant healthcare platforms where a single missed traversal in a document download endpoint means a breach notification — this pattern should be terrifying. A model that misses two of three resource-limit bugs on a 200-line file is going to miss far more on a 20,000-line service. Prediction: expect the next wave of AI security benchmarks to weight app-like fixtures far more heavily than snippet fixtures, and expect model vendors to fine-tune specifically for exhaustive sink enumeration rather than exemplar-first reasoning.

FAQ

Q: What is Snyk VulnBench JS 1.0? A: It’s a benchmark from Snyk that ran 300 vulnerability-scanning sessions across 10 small JavaScript and Express fixture projects, using both Snyk Code SAST and five Claude model configurations. Each task was repeated five times to measure how consistently each configuration reported the same bugs across identical runs.

Q: Should teams stop using LLMs for security review? A: No, but they shouldn’t use LLMs alone. Snyk’s data shows models are strong at familiar exploit shapes like command injection, hardcoded credentials, and SSRF, and can occasionally surface real gaps that SAST rules miss. But they’re noisy on unmatched findings, weaker on enumeration of repeated sinks, and their queues shift between runs — so pair them with deterministic SAST for coverage.

Q: Does a higher-cost model mean better security coverage? A: Not based on this benchmark. According to Snyk, Claude Opus 4.7 Max cost 5.67x more per session than Claude Opus 4.6 Medium but scored lower on Snyk-reference F1 (68.8% vs 75.4%) and produced far more one-off unmatched findings. Benchmark configurations against your own codebase before assuming newer or bigger equals better.

Key Takeaways

• Treat any LLM-reported vulnerability that isn’t reproduced across multiple internal runs as low-confidence — build sampling into your review pipeline before blocking merges on model findings.
• Deterministic SAST and agentic review have different blind spots; teams that ship both in a single review pipeline will out-triage teams that pick one.
• Benchmark model configurations on your own app-like code, not on snippet-sized fixtures — the 40.0% F1 collapse on Snyk’s largest fixture is the pattern you’ll actually hit in production.
• Expect AI security vendors to ship per-finding confidence scores derived from repeated internal sampling within the next product cycle; if your current tool doesn’t, ask when it will.
• The market is heading toward tiered AI reviewers where cheaper, more stable models beat larger, noisier ones for CI-scale security workloads — plan procurement accordingly.