Security · Comparison
Security scanners compared: accuracy, speed and cost
Supercov, Codex CLI, Claude Code, Kolega, DeepSeek, GLM and Semgrep on 72 vulnerable repositories: how much each finds, how fast, and what it costs.
The short version
- Most accurate: Codex CLI with GPT Daybreak Blue (F1 0.71) and Kolega DevSec Max (finds 91%). Several minutes and about $4 a repository.
- Cheapest AI scanner: a DeepSeek V4 Flash agent. 2¢ a repository, F1 0.48, 2.6 minutes.
- Fastest AI scanner: Supercov. 10 seconds and 5¢ a repository, F1 0.42.
- Rules only: Semgrep’s default rules find 9% of these vulnerabilities, in 8 seconds for free.
Results
Each scanner ran on the same 72 deliberately vulnerable repositories, with 2,016 labelled vulnerabilities between them.
| Scanner | F1 | Finds | Right | Speed | Cost per repository |
|---|---|---|---|---|---|
| Codex CLI, GPT Daybreak Blue | 0.71 | 82% | 63% | Slow, 5.4 min | $3.78 |
| Codex CLI, GPT-5.6 Sol | 0.70 | 79% | 62% | Slow, 8.8 min | $3.87 |
| Kolega DevSec Max 0.1.0 | 0.69 | 91% | 56% | not recorded | not recorded |
| DeepSeek V4.1 Flash agent | 0.51 | 50% | 53% | Minutes, 1.6 min | not recorded |
| GLM-5.3 agent | 0.51 | 54% | 48% | Slow, 7.9 min | 74¢ |
| Claude Code, Sonnet 5 | 0.49 | 44% | 57% | Minutes, 2.8 min | 95¢ |
| Codex CLI, GPT-6 Astra | 0.49 | 54% | 45% | Slow, 6.4 min | $6.99 |
| DeepSeek V4 Flash agent | 0.48 | 41% | 59% | Minutes, 2.6 min | 2¢ |
| DeepSeek V4 Pro agent | 0.46 | 37% | 63% | Minutes, 3.8 min | 7¢ |
| Supercov 2.0.1 | 0.42 | 43% | 42% | Instant, 10 s | 5¢ |
| Semgrep, auto rules | 0.11 | 9% | 14% | Instant, 8 s | free |
Finds is the share of labelled vulnerabilities found (recall). Right is the share of findings that were real (precision). F1 combines the two. Speed is the median time per repository: instant is under a minute, slow is over five.
Pick by what you need
| You want | Pick |
|---|---|
| The most findings before a release | Codex CLI with GPT Daybreak Blue, or Kolega |
| An AI review for pennies, and a few minutes to wait | A DeepSeek V4 Flash agent |
| A check your agent can run after every change | Supercov |
| Known patterns, free and offline | Semgrep, with rules for your stack |
Most teams will want two: something fast on every change, and a thorough scan before a release.
Compared one by one
- Supercov vs Codex CLI
- Supercov vs Claude Code
- Supercov vs Kolega
- Supercov vs DeepSeek agents
- Supercov vs Semgrep
How we measured
The repositories are the 72 in the half of RealVuln we did not tune Supercov on. They are Python, TypeScript and JavaScript apps written to be vulnerable, so expect different numbers on production code. Supercov 2.0.1 ran with default settings on a laptop. Every other scanner’s findings, time and cost come from RealVuln’s published runs, made on other machines, except Semgrep’s time, which we measured on the same laptop. Kolega builds RealVuln.
Accuracy, speed and cost has the details: what Supercov finds by vulnerability type, results by language, and how to rerun it.
Evidence: every scanner's results (JSON) · Supercov's reports, timings and scoring script (ZIP). Measured September 25, 2026 on 72 RealVuln repositories.