Mutation Testing

Definition: Mutation testing is a software testing technique where the source code is intentionally modified (mutated) to create faulty versions (mutants) of the tests of a program.

Purpose: To evaluate the effectiveness of test suite by determining if the tests can detect the introduced faults, thereby ensuring high test coverage and identifying weak spots in testing. (Testing the Tests)

Creation of Mutants: Mutants are the modified versions of the source code. They typically involve small changes, like replacing operators, modifying conditionals, or deleting lines of code.

Evaluation: When a test case fails on a mutant (i.e., the mutant produces different output or behavior than the original program), it is said to have been "killed." This indicates that the test case has detected the mutation, and it is considered a positive outcome. If a mutant is not detected by any test case, it is considered "surviving." This suggests that there might be an issue with the test suite because it failed to identify a potentially problematic mutation.

Outcome: The mutation score is the ratio of killed mutants to the total number of mutants. A higher score indicates a more effective test suite, while surviving mutants highlight areas where testing can improve.

Example:

Code:
def add(a, b): return a + b

Tests:
def test_add1(): assert add(0, 0) == 0

Mutated Code (change + to -):
def add(a, b): return a - b

In this case, the mutant will pass the test and survive.
We therby know that we need to improve our tests, for example by adding a second test case:
assert add(3, 5) == 8

Challenges

• For projects with a large code basis mutation testing can take a lot of time and computational resources. In such cases, one might schedule regular mutation testing runs, such as nightly or weekly. Some projects may choose to run mutation tests more frequently on the most critical or frequently changing parts of the codebase. This selective approach ensures that the most important areas are well-covered.
• Stubborn Mutation: The mutants in this set can be killed. Each stubborn mutant does have a test input that distinguishes its behaviour from that of the original program. However, none of these distinguishing test inputs has yet been found.
• Equivalent Mutation: The mutants in this set cannot be killed because they are equivalent to the original program. No possible test input exists that can distinguish their behaviour from that of the original program.
• Further information on stubborn and equivalent mutations can be found in this paper: Xiangjuan Yao, Mark Harman, and Yue Jia. 2014. A study of equivalent and stubborn mutation operators using human analysis of equivalence. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 919–930. https://doi.org/10.1145/2568225.2568265