Let's start this new year strong. Stryker 0.18 adds coverage analysis support for transpilers, starting with the TypeScript transpiler.
This article first explains the basics of (code) coverage analysis for mutation testing in general, before focussing on the new feature.
See the original PR for even more details:
It might not sound particularly sexy, but (code) coverage analysis is an important feature of any mutation testing framework. Mutation testing takes a long time. The most effective way to save time, is to do less work.
As you can see the
product functions are reasonably well tested, but the the
difference function isn't tested at all.
For example: what would the result be for the mutant that changes
a - b to
a + b? It doesn't even make sense to test it,
anyone can see it will survive anyway.
If you set coverage analysis to
'all' inside your stryker.conf.js file, this is exactly what happens. Stryker will automatically
collect code coverage results during the initial test run phase*. If a mutant mutates code that is not tested,
it will be marked as 'no coverage' (which translates to survived during the score calculation) without ever testing it.
This can safe minutes on larger code bases.
- Coverage analysis is not supported for Jest yet.
Nice! We're already saving time by analyzing a simple code coverage result. But if we take a closer look, we see that we can save even more time.
product function for example: we want to test the mutant that changes
a * b to
a / b. We see that it is covered by a test, so
we cannot skip the actual testing of this mutant. However, we can save time by only running the one test that covers this mutant.
If you set coverage analysis to
'perTest' inside your stryker.conf.js file, this is exactly what happens. Stryker will automatically
collect code coverage results per test during the initial test run phase. Next, it will select only those tests that actually cover
a mutant to run for that mutant (again, not supported for Jest yet). This might seem like a small improvements,
but in big projects with 100s of tests, it quickly adds up to minutes.
It is important to realize that this does not influence the quality of the resulting report at all. It simply less work to reach the same conclusion.
Now that we have a firm understanding of coverage analysis and why it is important, lets dive into the new stuff. After all, coverage analysis is nothing new for Stryker, it has been there for a long time. However, it gets more complicated when you add a transpiler to the (delicious) mutation testing mix.
Please read our blog article on transpiling code if your interested in the details on how transpilers are implemented in Stryker. For now it is is enough that you understand that Stryker mutates your source code, while running the transpiled code in the test runner. Any coverage analysis we collect will only tell us about the transpiled covered code, while the mutants only work on your source code.
Take this small piece of TypeScript code:
Even with this small example, you can already see that the location of
a + b in source code vs transpiled code is totally different.
We need to translate the location of a mutant to the actual transpiled location somehow.
Luckily, any respectable transpiler can produce source maps. They are used by code editors and browsers to debug source code while executing transpiled/minified/bundled code. And, as of now, this is what is used by Stryker to calculate the transpiled location.
So if you select coverage analysis 'all' or 'perTest' while transpiling using typescript, the stryker-typescript transpiler plugin will produce source maps during the initial test run. Stryker will use those source maps to translate the transpiled location of all mutants.
Unfortunately, we do not have a good performance testing mechanism for Stryker yet (PRs and ideas are welcome!), but the results for mutation testing on Stryker itself are very promising, shaving off about 6 minutes for single run, about a 50% performance increase.
Next up we'll work on getting this performance increase to all supported transpilers. We're also working behind the scenes on a transpiler plugin for webpack. Which you can also expect within the next couple of weeks.
In the meantime: have fun with this feature. Are you using it? Please let is know what you think! We're always interested to know what we can improve further.