# Mutation testing AI-generated tests: 100% coverage, up to 22% of bugs missed

> We ran Stryker on 24 test suites, 20 written by Claude Code. Most covered every line, yet they caught 78% to 99% of planted bugs. What survived and why.

Web page: https://supercov.com/blog/mutation-testing-ai-generated-tests · Published 2026-10-24
All Supercov pages for agents: https://supercov.com/llms.txt · Full text: https://supercov.com/llms-full.txt

Claude Code's suites covered every line. Mutation testing showed which ones would notice a bug.

Claude Code wrote 20 test suites for four TypeScript libraries. 19 of them
covered every line. Then we ran mutation testing on all of them, and on the
maintainers' own tests.

The suites that covered every line caught between 78% and 99% of the bugs
Stryker planted. Coverage rated them the same. Mutation testing did not, and
on these libraries it took 3 to 16 seconds a run.

## What mutation testing measures

A mutation tester changes your source in small ways, one change at a time:
`<` becomes `<=`, a condition becomes `true`, a string becomes empty. Each
change is a mutant. It runs your tests against every mutant. If a test fails,
the mutant is killed. If every test still passes, it survived: the tests run
that code without checking what it does.

The mutation score is the share of mutants killed. Line coverage asks whether
a line ran. Mutation score asks whether a test would notice it being wrong.

## 24 suites, the same coverage, different scores

The libraries are four from [unjs](https://github.com/unjs): destr, defu,
scule and cookie-es. Claude Code wrote each suite in a fresh copy with the
tests removed, five ways: with no skill, with three popular testing skills and
with the Supercov plugin. The maintainers' own tests make four more suites.

| Library | Mutants | Line coverage, Claude Code's suites | Mutation score, Claude Code's suites | Maintainers' tests |
| --- | ---: | ---: | ---: | ---: |
| destr | 141 | 100% | 78.0–93.6% | 78.7% |
| defu | 103 | 97.4–100% | 91.3–98.1% | 84.5% |
| scule | 78 | 100% | 93.6–98.7% | 92.3% |
| cookie-es | 725 | 100% | 86.3–92.1% | 86.9% |

![Mutation score of each suite by library. destr: Claude Code's suites 78.0% to 93.6%, maintainers 78.7%. defu: 91.3% to 98.1%, maintainers 84.5%. scule: 93.6% to 98.7%, maintainers 92.3%. cookie-es: 86.3% to 92.1%, maintainers 86.9%. Every Claude Code suite covered 97 to 100% of lines.](https://supercov.com/assets/mutation-testing-ai-suites-1360.667442a51d.webp)

*The share of Stryker's mutants each suite killed. Line coverage was the same for nearly all of them.*

On average Claude Code's suites caught 92.1% of mutants, more than the
maintainers' 85.6%. The spread inside each library is the point: suites with
the same coverage left very different numbers of bugs uncaught.

## What survived

Across Claude Code's 20 suites, 487 mutants survived. Most were small edits
to logic:

| Change | Share of survivors |
| --- | ---: |
| A condition replaced with `true` or `false` | 37% |
| A comparison changed, such as `<` to `<=` | 20% |
| A regular expression changed | 17% |
| A string replaced with an empty one | 12% |

Some survivors are real gaps. destr drops a `constructor` key to prevent
prototype pollution:

```ts
if (
  key === "__proto__" ||
  (key === "constructor" &&
    value &&
    typeof value === "object" &&
    "prototype" in value)
) {
```

Replace `key === "constructor"` with `true` and every test still passes. No
suite, the maintainers' included, parses an ordinary key whose value has a
`prototype` property, so none notices that the guard would drop it.

Others cannot change behaviour. destr also checks the length of a string
before comparing it with keywords such as `"infinity"`:

```ts
if (_value.length <= 9) {
  switch (_value.toLowerCase()) {
```

Make that condition `true` and the result is the same, because no longer
string matches a case. The check only saves time. A mutant like this is
equivalent: no test can kill it. 57 mutants survived every one of the 24
suites, and several are of this kind. That is why 100% is rarely the target.

## What mutation testing costs

Stryker reruns the tests once per mutant, so it costs more than a test run.
Here it ran with per-test coverage analysis and four workers:

| Library | Test run | Stryker run |
| --- | ---: | ---: |
| destr | 1.2–1.8 s | 4.7–6.6 s |
| defu | 1.0–1.8 s | 3.3–4.3 s |
| scule | 1.1–2.1 s | 3.2–3.5 s |
| cookie-es | 1.2–1.7 s | 14.0–16.0 s |

These are small libraries. The cost grows with the number of mutants and the
tests that cover each one, so on a large codebase run it on changed files, or
use Stryker's `--incremental` mode, which reuses results for code that did not
change.

## Run it on what your agent writes

Pick the tool for your language:

```sh
npx stryker init && npx stryker run        # JavaScript and TypeScript
mutmut run && mutmut results                # Python
mvn org.pitest:pitest-maven:mutationCoverage  # Java and Kotlin, PIT
cargo mutants                               # Rust
```

Then give the survivors back to the agent with a clear rule:

```text
Read the surviving mutants in the mutation report. For each one, add or
change a test so it fails on the mutant and passes on the original code.
Only change tests. If a mutant cannot change behaviour, say why and skip it.
Run the mutation tester again and report the score before and after.
```

The last instruction matters. An agent told to kill every mutant will write
odd tests for equivalent ones.

## A faster first check: assertion coverage

[Supercov](https://supercov.com) has a related measurement,
[assertion coverage](https://supercov.com/docs/assertions.md). It asks mutation testing's question
one statement at a time: would a test fail if this statement changed? A value
becomes `undefined`, a condition is inverted, a `return true` becomes
`return false`. Instead of running each change, Jev reads the code and the
tests that ran it and judges the answer.

```sh
npx supercov -- npm test
npx supercov runs latest assertions assess
npx supercov runs latest assertions
```

On the same 24 suites it took 1 to 6 seconds a suite and $0.07 in total. It
judged 2,130 of 2,144 executed statements asserted. That fits the mutation
results: these suites check what each statement produces, and what they miss
are finer edits such as `<` to `<=`.

Use assertion coverage first, to find code your tests run without checking at
all. Use mutation testing where the details matter. More on the difference in
[assertion coverage without mutation testing](https://supercov.com/blog/assertion-coverage-without-mutation-testing.md).

## How we ran it

- destr, defu, scule and cookie-es at pinned commits, with their tests and
  git history removed. Claude Code 2.1.280 with `claude-sonnet-5-5` wrote one
  suite per library five ways, with web access off.
- Every suite ran on the library's original source. Line coverage is Vitest's
  V8 provider. Mutation score is Stryker's (10.0, and 8.7 for scule): killed
  and timed-out mutants over all valid ones.
- Assertion coverage is Supercov 4.19.0 with Jev.
- One suite per library and setup, so a point or two between suites is noise.

[Download every suite, Stryker report and script](https://supercov.com/downloads/supercov-mutation-testing-ai-tests.zip),
or the [results as JSON](https://supercov.com/downloads/mutation-testing-ai-tests-2026-10-24.json).
For what line and branch coverage do measure, see
[line coverage vs branch coverage](https://supercov.com/compare/line-vs-branch-coverage.md).