Mutation testing AI-generated tests: 100% coverage, up to 22% of bugs missed

Claude Code wrote 20 test suites for four TypeScript libraries. 19 of them covered every line. Then we ran mutation testing on all of them, and on the maintainers’ own tests.

The suites that covered every line caught between 78% and 99% of the bugs Stryker planted. Coverage rated them the same. Mutation testing did not, and on these libraries it took 3 to 16 seconds a run.

What mutation testing measures

A mutation tester changes your source in small ways, one change at a time: < becomes <=, a condition becomes true, a string becomes empty. Each change is a mutant. It runs your tests against every mutant. If a test fails, the mutant is killed. If every test still passes, it survived: the tests run that code without checking what it does.

The mutation score is the share of mutants killed. Line coverage asks whether a line ran. Mutation score asks whether a test would notice it being wrong.

24 suites, the same coverage, different scores

The libraries are four from unjs: destr, defu, scule and cookie-es. Claude Code wrote each suite in a fresh copy with the tests removed, five ways: with no skill, with three popular testing skills and with the Supercov plugin. The maintainers’ own tests make four more suites.

On average Claude Code’s suites caught 92.1% of mutants, more than the maintainers’ 85.6%. The spread inside each library is the point: suites with the same coverage left very different numbers of bugs uncaught.

What survived

Across Claude Code’s 20 suites, 487 mutants survived. Most were small edits to logic:

Some survivors are real gaps. destr drops a constructor key to prevent prototype pollution:

if (
  key === "__proto__" ||
  (key === "constructor" &&
    value &&
    typeof value === "object" &&
    "prototype" in value)
) {

Replace key === "constructor" with true and every test still passes. No suite, the maintainers’ included, parses an ordinary key whose value has a prototype property, so none notices that the guard would drop it.

Others cannot change behaviour. destr also checks the length of a string before comparing it with keywords such as "infinity":

if (_value.length <= 9) {
  switch (_value.toLowerCase()) {

Make that condition true and the result is the same, because no longer string matches a case. The check only saves time. A mutant like this is equivalent: no test can kill it. 57 mutants survived every one of the 24 suites, and several are of this kind. That is why 100% is rarely the target.

What mutation testing costs

Stryker reruns the tests once per mutant, so it costs more than a test run. Here it ran with per-test coverage analysis and four workers:

These are small libraries. The cost grows with the number of mutants and the tests that cover each one, so on a large codebase run it on changed files, or use Stryker’s --incremental mode, which reuses results for code that did not change.

Run it on what your agent writes

Pick the tool for your language:

npx stryker init && npx stryker run        # JavaScript and TypeScript
mutmut run && mutmut results                # Python
mvn org.pitest:pitest-maven:mutationCoverage  # Java and Kotlin, PIT
cargo mutants                               # Rust

Then give the survivors back to the agent with a clear rule:

Read the surviving mutants in the mutation report. For each one, add or
change a test so it fails on the mutant and passes on the original code.
Only change tests. If a mutant cannot change behaviour, say why and skip it.
Run the mutation tester again and report the score before and after.

The last instruction matters. An agent told to kill every mutant will write odd tests for equivalent ones.

A faster first check: assertion coverage

Supercov has a related measurement, assertion coverage. It asks mutation testing’s question one statement at a time: would a test fail if this statement changed? A value becomes undefined, a condition is inverted, a return true becomes return false. Instead of running each change, Jev reads the code and the tests that ran it and judges the answer.

npx supercov -- npm test
npx supercov runs latest assertions assess
npx supercov runs latest assertions

On the same 24 suites it took 1 to 6 seconds a suite and $0.07 in total. It judged 2,130 of 2,144 executed statements asserted. That fits the mutation results: these suites check what each statement produces, and what they miss are finer edits such as < to <=.

Use assertion coverage first, to find code your tests run without checking at all. Use mutation testing where the details matter. More on the difference in assertion coverage without mutation testing.

How we ran it

• destr, defu, scule and cookie-es at pinned commits, with their tests and git history removed. Claude Code 2.1.280 with claude-sonnet-5-5 wrote one suite per library five ways, with web access off.
• Every suite ran on the library’s original source. Line coverage is Vitest’s V8 provider. Mutation score is Stryker’s (10.0, and 8.7 for scule): killed and timed-out mutants over all valid ones.
• Assertion coverage is Supercov 4.19.0 with Jev.
• One suite per library and setup, so a point or two between suites is noise.

Download every suite, Stryker report and script, or the results as JSON. For what line and branch coverage do measure, see line coverage vs branch coverage.