Skip to main content

Announcing Stryker.NET 1.0

Β· 9 min read
Richard Werkman

We're proud to announce the first major release of Stryker.NET: 1.0. It comes with exciting new features and an overhaul of how you interact with the Stryker CLI. With the release of Stryker 1.0, we wanted to create a cohesive and intuitive user experience.

If you're new to mutation testing, it's a way to measure your tests' effectiveness. A mutation testing framework will make small changes, called mutants, one by one in your source code. Then it will run your tests to see if one of them fails. If so, you just "killed" that mutant; if not, it "survived". If too many mutants survive, you probably want to improve your tests. The mutation testing report will give you insides into the test cases you may have missed. If this all sounds complicated, please take a look at our RoboBar πŸ€–πŸ· example.

If you're new to Stryker.NET, please follow our Getting started guide. Are you already using Stryker.NET? Update to the latest version with the following command:

Global install:

dotnet tool update dotnet-stryker --global

Project install:

dotnet tool update dotnet-stryker --local

With that out of the way, let's dive into the new stuff!

πŸ’₯ Breaking changes

⏭ Updated runtime

The .NET runtime for Stryker.NET has been updated from 3.1 to 5.0.

This update should have some performance benefits, and it will help with developing Stryker.NET in the future.

Please download and install .NET 5.0 or update your pipeline to support .NET 5.

Note that you do not have to update your application to use dotnet 5. Dotnet 5 is only a runtime requirement for Stryker to be able to run on your system.

⏭ Options rework

Most options have been renamed or work differently.

A fundamental difference is how you pass multi-value options from the command line.

Multi value options

The old annotation for passing multi-value options was confusing and not based on any standards. For example, this is how you provided multiple reporters pre 1.0 πŸ‘Ž

dotnet stryker --reporters "['html', 'progress']"

From 1.0 onward, this now looks like πŸ‘

dotnet stryker --reporter "html" --reporter "progress"

Options migration guide

Most options have a new name. We have also decided that some options either don't belong on the commandline or don't belong in the configuration file. For example, an API key should not be stored in the configuration file so that possibility has been removed.

Options migration overview:

Old cliNew cliOld jsonNew json
config-file-pathf | config-file❌❌
max-concurrent-testrunnersc | concurrencymax-concurrent-testrunnersconcurrency
dev-modedev-modedev-mode❌
solution-paths | solutionsolution-pathsolution
log-fileL | log-to-filelog-file❌
log-levelV | verbositylog-levelverbosity
mutation-levell | mutation-levelmutation-levelmutation-level
threshold-high❌thresholds.highthresholds.high
threshold-low❌thresholds.lowthresholds.low
threshold-breakb | break-atthresholds.breakthresholds.break
reportersr | reporter (flag allowed multiple times)reportersreporters
project-filep | projectproject-fileproject
diffsincediffsince
timeout-ms❌timeout-msadditional-timeout
excluded-mutations❌excluded-mutationsignore-mutations
ignore-methods❌ignore-methodsignore-methods
mutatem | mutatemutatemutate
language-version❌language-versionlanguage-version
coverage-analysis❌coverage-analysiscoverage-analysis
abort-test-on-fail❌abort-test-on-faildisable-bail
disable-testing-mix-mutations❌disable-testing-mix-mutationsdisable-mix-mutants
test-projects❌test-projectstest-projects
dashboard-url❌dashboard-urldashboard-url
dashboard-api-keydashboard-api-keydashboard-api-key❌
project-name❌dashboard-projectproject-info.name
module-name❌dashboard-moduleproject-info.module
project-versionv | versiondashboard-versionproject-info.version
diff-ignore-files❌diff-ignore-filessince.ignore-changes-in
azure-storage-url❌azure-storage-urlbaseline.azure-fileshare-url
dashboard-fallback-version❌dashboard-fallback-versionbaseline.fallback-version
baseline-storage-location❌baseline-storage-locationbaseline.provider
dashboard-comparewith-baselinedashboard-comparebaseline
git-diff-target--since ...git-diff-targetsince.target
azure-storage-sasazure-fileshare-sasazure-storage-sas❌
files-to-exclude❌❌❌
test-runner❌❌❌

❌ means the option has been removed.

πŸš€ What's new?

This release comes packed with new features! Let's walk through them all:

Statement removal mutator

We introduced a new mutator that removes statements that otherwise would have been untouched by other mutators.

The mutator will remove the following statements:

  • return
  • break
  • continue
  • goto
  • throw
  • yield return
  • yield break
  • expression

Allow failing tests

It's now allowed to start a mutation test run even with failing tests. Stryker will try to make the best of the situation by marking mutants covered by initially failing tests as survived.

Mutant filtering

It's now possible to filter mutants at the source code level using special comments. This filtering gives the most fine-grained level of control.

The syntax for the comments is: Stryker [disable|restore][once][all| mutator list][: reason for disabling]

// Stryker disable all Disables all mutants from that line on.

// Stryker restore all re-enables all mutants from that line on.

// Stryker disable once all will only disable mutants on the next line.

// Stryker disable once Arithmetic,Update will only disable Arithmetic and Update mutants on the next line

Example:

var i = 0;
var y = 10;
// Stryker disable all : for explanatory reasons
i++; // won't be mutated
y++; // won't be mutated
// Stryker restore all
i--; // will be mutated
// Stryker disable once all
y--; // won't be mutated
i++; // will be mutated
// Stryker disable once Arithmetic
y++; // will be mutated
// Stryker disable once Arithmetic,Update
i--; // won't be mutated

Note that this feature is scope aware. If you disable mutators inside a method, the scope will not leak outside the method, even if there is more code below.

Ignore mutations

The ignore-mutations option now offers more fine-grained control. Before v1.0 it was possible to ignore complete mutators. Now it's possible to ignore specific mutants inside these mutators as well.

Example:

"stryker-config": {
"ignore-mutations": [
"linq.First",
"linq.Sum"
]
}

Note: this only works for Linq mutations for now, but we plan to bring this functionality to all mutations.

MsBuild path option

By default Stryker tries to auto-discover MSBuild on your system. However, if Stryker fails to discover the correct MSBuild, you may manually supply the path with this option.

Example:

--msbuild-path "c://MsBuild/MsBuild.exe"

Target framework

If the project targets multiple frameworks, it is now possible to specify the particular framework to build against. If you set a non-existent target, Stryker will build the project against a random one (or the only one if so).

Example:

{
"stryker-config": {
"target-framework": "netcoreapp3.1"
}
}

Filter test cases

A long-awaited feature has finally found its way into Stryker! It is now possible to exclude some test cases. So, for example, if you have long-running integration tests in your unit test project, they can be disabled for Stryker, improving the performance.

Example:

{
"stryker-config": {
"test-case-filter": "(FullyQualifiedName~UnitTest1&TestCategory=CategoryA)|Priority=1"
}
}

Uses dotnet test --filter option syntax, detailed here.

Filling all settings to use the dashboard reporter could be a bit of a hassle. However, thanks to Source Link, the repository URL and the full version (including the git SHA1) of a project can be included in the produced assembly.

Stryker now uses the information computed by SourceLink to automatically retrieve the project name (github.com/organization/project) and project version, both of which are requirements for the dashboard reporter.

Enable this by adding the following to your .csproj:

  <ItemGroup>
<PackageReference Include="DotNet.ReproducibleBuilds" Version="0.1.66" PrivateAssets="All"/>
</ItemGroup>

For more information on SourceLink and ReproducibleBuilds see SourceLink and Dotnet.ReproducibleBuilds

Block removal mutations

Finally the last missing "common" mutation is added to Stryker.NET! Block removal mutations empty every block statement in your code. This means method bodies or statement bodies (if, while, for). With this mutation Stryker will have improved coverage on your complete codebase. Every method will have at least one mutation and thus should have at least one test.

πŸ› Bug fixes

No more mutated assembly on disk after Stryker run

During mutation testing, Stryker replaces your system under test assembly on disk. Up till now, the mutated assembly stayed in place after mutation testing was done. This had some unintended side effects. For example, code coverage results could be incorrect until you rebuild your project, and there was the risk of accidentally releasing/publishing the mutated assembly instead of the original if you did not rebuild your project after mutation testing in your pipelines. We now copy your original assembly before we modify it and place it back after mutation testing. No more rebuild required!

πŸ‘ͺ Team expansion

We welcome Cyrille DUPUYDAUBY to the team! They have supported us with feedback, testing, and development since 2018. It was long past due that we officially recognized their contributions to the project!

πŸŽ‰ Thank you

Special thanks to Cyrille DUPUYDAUBY, Peter Semkin, Philip Rowan, Corentin Altepe, CΓ©dric Luthi, Gregory Bell, John McGlynn, Beatrice Forslund, dukedagmor and anyone else we may have missed for their efforts to make this release happen πŸ‘ We truly appreciate all the help and feedback we receive!