# Accuracy, speed and cost

> How often Supercov finds real vulnerabilities, how fast, and what it costs, next to ten other scanners.

Web page: https://supercov.com/docs/security-benchmark
All Supercov pages for agents: https://supercov.com/llms.txt · Full text: https://supercov.com/llms-full.txt

We ran `npx supercov security` on 72 repositories with known, labelled
vulnerabilities and scored it next to ten other scanners on the same code.

In short: Supercov finds 43% of the labelled vulnerabilities in a median of
10 seconds, for about 5 cents a repository. The strongest AI agents find about
80%, but take 5 to 9 minutes and cost about $4.

## Results

72 repositories from [RealVuln](https://github.com/kolega-ai/Real-Vuln-Benchmark),
2,016 labelled vulnerabilities, scored with the benchmark's own rule.

| Scanner | F1 | Precision | Recall | Median time | Cost per repository |
| --- | ---: | ---: | ---: | ---: | ---: |
| Codex CLI, GPT Daybreak Blue | 0.71 | 63% | 82% | 5.4 min | $3.78 |
| Codex CLI, GPT-5.6 Sol | 0.70 | 62% | 79% | 8.8 min | $3.87 |
| Kolega DevSec Max 0.1.0 | 0.69 | 56% | 91% | not recorded | not recorded |
| DeepSeek V4.1 Flash agent | 0.51 | 53% | 50% | 1.6 min | not recorded |
| GLM-5.3 agent | 0.51 | 48% | 54% | 7.9 min | 74¢ |
| Claude Code, Sonnet 5 | 0.49 | 57% | 44% | 2.8 min | 95¢ |
| Codex CLI, GPT-6 Astra | 0.49 | 45% | 54% | 6.4 min | $6.99 |
| DeepSeek V4 Flash agent | 0.48 | 59% | 41% | 2.6 min | 2¢ |
| DeepSeek V4 Pro agent | 0.46 | 63% | 37% | 3.8 min | 7¢ |
| **Supercov 2.0.1** | **0.42** | **42%** | **43%** | **10 s** | **5¢** |
| Semgrep, auto rules | 0.11 | 14% | 9% | 8 s | free |

- **Recall** is the share of labelled vulnerabilities a scanner found.
- **Precision** is the share of its findings that were real.
- **F1** combines the two into one number.

Supercov is the fastest AI-based scanner by a wide margin, and one of the
cheapest. Only Semgrep, which matches rules instead, is faster. Supercov is also
the least accurate of the AI-based scanners.

## What it finds, and what it misses

| Vulnerability | Found |
| --- | ---: |
| Open redirect, server-side request forgery | 89% |
| SQL, command and other injection | 86% |
| Unsafe code execution | 82% |
| Path traversal | 79% |
| Mass assignment | 64% |
| Cross-site scripting | 56% |
| Hard-coded secrets | 53% |
| Weak cryptography | 48% |
| Sensitive data exposure | 43% |
| Insecure configuration | 43% |
| Weak authentication | 24% |
| Missing authorization | 11% |

It is good at problems visible on one line: a query built from user input, a
path taken from a request. It misses most authorization and authentication
problems, because the check that should protect a route usually lives in
another file. 269 of the labels are kinds Supercov does not check for at all,
such as missing rate limits.

It also reports the most false positives on the benchmark's decoy findings:
26, against 1 to 23 for the others.

## By language

| Language | Repositories | Supercov F1 | Best F1 |
| --- | ---: | ---: | ---: |
| Python | 32 | 0.54 | 0.82 |
| TypeScript | 27 | 0.36 | 0.65 |
| JavaScript | 13 | 0.34 | 0.74 |

Python is its strongest language. On JavaScript, many of its false positives
are in bundled third-party libraries, such as copies of jQuery or Bootstrap,
that it should skip.

## When to use something else

- **You want the most findings and can wait.** A Codex CLI agent or Kolega
  finds nearly twice as much. Run it before a release, not on every change.
- **You want low cost and can wait a few minutes.** A DeepSeek V4 Flash agent
  is cheaper and more accurate, at about 2.6 minutes a repository.
- **You want fast feedback on every change.** Supercov answers in seconds, so
  your agent can run it after each edit with [`npx supercov security patch`](https://supercov.com/docs/security.md).

The tools also combine well: Supercov on every change, an agent-based scan
before a release.

## How we measured

- **Benchmark:** 72 of RealVuln's 140 repositories, the half we did not tune
  the checks on, each at the commit its labels were written for. They are
  deliberately vulnerable Python, TypeScript and JavaScript apps, so results on
  production code will differ.
- **Supercov:** version 2.0.1 with its default settings, one repository at a
  time on a laptop. Cost is Jev's list price of $42 per billion input tokens.
- **Other scanners:** their findings, times and costs are RealVuln's own
  published runs, and their times come from different machines. We timed
  Semgrep ourselves, with the same version and settings, on the laptop that ran
  Supercov. GLM-5.3 had 3 failed runs, counted as finding nothing.
- **Scoring:** a finding counts when it names the right file, a matching
  vulnerability type and a line within 10 of the label.

Download the [results](https://supercov.com/downloads/security-benchmark-2026-09-25.json) or
[everything needed to rerun it](https://supercov.com/downloads/supercov-security-benchmark-2026-09-25.zip):
Supercov's report for every repository, the times and the scoring script.
Checked September 25, 2026.

See also [Security scanners compared](https://supercov.com/compare/security.md).