This page should shed some light on the different outcomes a mutant can have and the different metrics you'll find in any given mutation testing report.
A mutant can have one of the following states:
- Killed When at least one test failed while this mutant was active, the mutant is killed. This is what you want, good job!
- Survived When all tests passed while this mutant was active, the mutant survived. You're missing a test for it.
- No coverage No tests were executed for this mutant. It probably is located in a part of the code not hit by any of your tests. This means the mutant survived and you are missing a test case for it.
- Timeout The running of tests with this mutant active resulted in a timeout. For example, the mutant resulted in an infinite loop in your code. Don't spend too much attention to this mutant. It is counted as "detected". The logic here is that if this mutant were to be injected in your code, your CI build would detect it because the tests will never complete.
- Runtime error
The running of the tests resulted in an error (rather than a failed test).
This can happen when the testrunner fails. For example, when a testrunner throws an
OutOfMemoryErroror for dynamic languages where the mutant resulted in unparsable code. Don't spend too much attention looking at this mutant. It is not represented in your mutation score.
- Compile error The mutant resulted in a compiler error. This can happen in compiled languages. Don't spend too much attention looking at this mutant. It is not represented in your mutation score.
- Ignored The mutant was not tested because the config of the user asked for it to be ignored. This will not count against your mutation score but will show up in reports.
Based on these states, we can calculate the following metrics:
killed + timeout
The number of mutants detected by your tests.
survived + no coverage
The number of mutants that are not detected by your tests.
detected + survived
The number of mutants that your tests produce code coverage for.
detected + undetected
The number of valid mutants. They didn't result in a compile error or runtime error.
runtime errors + compile errors
The number of invalid mutants. They couldn't be tested because they produce either a compile error or a runtime error.
- Total mutants
valid + invalid + ignored
- Mutation score
detected / valid * 100
The total percentage of mutants that were killed.
- Mutation score based on covered code
detected / covered * 100
The total percentage of mutants that were killed based on the code coverage results.