What is Mutation Testing

Every developer writes unit tests, and much of the reporting on unit testing is around code coverage, with tools such as SonarQube incorporated in build pipelines to enforce a targeted percentage of lines of codes covered by tests. We write numerous tests, cover countless lines of code to get through this gate, but how do we really know how good these tests are, or if we're really testing anything at all?

How do we 'test the tests'?

This is where mutation testing comes in. Mutation testing, in its simplest form, is taking a test, manipulating the code being tested and then executing the test to see if it still passes. If the test fails, the mutant is said to have been killed, and if the test passes, the mutant deemed to have survived.

The great thing about mutation tests, is that because they work directly on our unit tests, we do not need to write any additional code, so there's not really an excude not to at least explore mutation testing. For test written in C# and .Net, we can use Stryker.NET to execute mutation tests, but versions of Stryker are available for JavaScript and Scala also.

A Quick Example

A Simple Test

Let's take the following piece of code and associated test:

public class VehicleCheck
{
    public bool IsMotorbike(bool hasEngine, int numberOfWheels)
    {
        if (hasEngine && numberOfWheels == 2)
            return true;
        return false;
    }
}


public class VehicleCheckTest
{
    [Fact]
    public void HasTwoWheelsAndEngine_IsMotorbike_Succeed()
    {
        var checker = new VehicleCheck();

        var isMotorbike = checker.IsMotorbike(true, 2);

        Assert.True(isMotorbike);
    }
}

If we run this test, it will pass.

What Does Stryker Do?

Stryker will go through the being tested and start to make changes based on a simple set of 'mutations' as outlined here.

For our test, Stryker will make a number of mutations to the code as follows :

• The && becomes ||
• The true becomes false
• The false becomes true
• the (hasEngine && numberOfWheels == 2) becomes !(hasEngine && numberOfWheels == 2)
• Delete the entire contents of the code block

For our example, we will only consider the first one:

public class VehicleCheck
{
    public bool IsMotorbike(bool hasEngine, int numberOfWheels)
    {
        if (hasEngine || numberOfWheels == 2)
            return true;
        return false;
    }
}

For each mutation, Stryker will then run the unit tests again, and for the example above the test will still pass.

So, What's the Problem?

The main purpose of unit tests is to highlight if someone makes a breaking change to our code. Stryker has made a significant change in the logic of the application, and yet all the tests still pass. This is the opposite of what our test suite should be designed to do.

Stryker will report this back as a survived mutant and therefore a failed mutation test.

Code coverage metrics will still report a high degree of coverage which is great, but Stryker is suggesting to us that the quality of our test scenarios could be improved, as we can currently make changes to the code logic, and they will still pass.

Here it's pretty clear now that if I ran the test against the mutated code claiming to have no engine and 2 wheels, or an engine and 4 wheels, I'd still pass the check for a motorbike, which is clearly incorrect.

Lets Kill the Mutant

Because Stryker requires that all unit tests will pass before running mutation tests, when Stryker makes a change and re-runs the related tests, only one test has to fail for the mutant to be deemed killed. For the above example, we could simply add a couple of additional tests to cover those aforementioned scenarios.

Let's add those tests:

public class VehicleCheckTest
{
    [Fact]
    public void HasTwoWheelsAndEngine_IsMotorbike_Succeed()
    {
        var checker = new VehicleCheck();

        var isMotorbike = checker.IsMotorbike(true, 2);

        Assert.True(isMotorbike);
    }


    [Fact]
    public void HasFourWheelsAndEngine_IsMotorbike_Fails()
    {
        var checker = new VehicleCheck();

        var isMotorbike = checker.IsMotorbike(true, 4);

        Assert.False(isMotorbike);
    }

    [Fact]
    public void HasTwoWheelsAndNoEngine_IsMotorbike_Fails()
    {
        var checker = new VehicleCheck();

        var isMotorbike = checker.IsMotorbike(false, 2);

        Assert.False(isMotorbike);
    }
}

You will actually notice that in the first run 5 mutants were counted and in the second 6 mutants were counted. In both instances, Stryker actually found 8 possible mutations, but there was no test coverage for all of them. This is something that could also be considered when reviewing the quality of the tests that have been created.

How Do I Run These Tests?

The simplest way to run tests with Stryker is to install Stryker globally

dotnet tool install -g dotnet-stryker

And then navigate to the directory where your unit test project is and run

dotnet stryker

HTML Reports

By default, Stryker also creates a nice HTML report output that allows you to view the output scores and review what was changed for each test, the below example being from the first test showing the survived mutation: