As the demand for software grows, so is the complexity of the design. The more complex a software is, the higher will be the testing needs, quality assurance and customer satisfaction. Although testing is, an integral part of the software development process the issue is still on what is sufficient or adequate testing still open.

If your tests cannot find a bug, can you believe that there are no bugs? Is it possible to trust your unit tests? Can you ascertain that they are telling truth for you? How does it work out to assess your tests? Mutation test is one of those wonderful tests that allow you to assess the tests. Mutation test involves deliberately altering, modifying or changing a program code, later rerunning a suit of correct unit tests against the mutated program.

In nutshell, mutation test works in the following manner. Initially, you will need to start the test with a piece of production code that the unit tests cover in the right way. Once you ascertain that all tests pass for a given code, and then you can start applying mutation to the target assembly.

Testing means additional cost and less productivity. For optimum development processing, there is a great need for the continuous improvement of quality testing methods. Testing coverage is an important issue in software development. Testing coverage is how well you test the software. The question is when do you stop testing? What is “adequate” testing? Right now, the most common practice is to state the requirement of percentage coverage like “a minimum xyz% coverage”. You will need to balance various requirements to bring about quality and customer satisfaction against productivity and cost.

Another important question to consider is, “How extensive are the testing programs?” What is the assurance for the “xyz% coverage” claim? What determines that at least “xyz%” of all possible causes of errors, bugs or failures is covered by the existing testing methods? Mutation testing is one way to verify if the software tester performed the testing in a proper manner. Through mutation testing, it can be determined, if the set of testing methods used in the developing process were appropriate and adequate to ensure product quality.

What is Mutation Testing?

To determine the correctness of a testing program, you will need to observe the behavior for each test case. If there are no detected faults, then the program is correct or it passes the test. A test case is the set of input values to the program under test and the corresponding output values.

Mutation testing is a method or strategy to check the effectiveness or accuracy of a testing program to detect the fault in the system or program under test. The test is called so because mutants of the software are created and run with the test cases.

Experts also call mutation testing as fault-based testing strategy because the mutants are created by introducing a single fault into the original program. You can set any number of mutants into the system. There are other versions of mutation testing, but they rely on the same method – to mutate the original software.

Some of the other versions are weak mutation, interface mutation and specification based mutation. Mutation testing is not a new method, developers know about it since the late 1970s. However, researchers have been using it in the educational institutions than in the industrial software domain.

Mutation Testing Procedure

The following steps rely on the traditional mutation process:

Create mutant software

The system /program under test are modified by rewriting the source code to introduce a single fault. The new versions of the system/program are mutants of the original ones. Each mutant has one fault, different from the original software and the other mutants. The tester may create as many mutants as needed. There is no limit.

Prepare test cases for input to the original and mutant software

The test cases should plan for the detection of the fault introduced into the mutant software. The tester may improve on the test cases, if necessary, so that the fault in the mutant software is found.

Apply test cases to original software

In the original software, the output is supposed to be correct, or must be similar as the expected output. If not, then there would be an error in the application, which must be fixed. Apply the test cases again until there are no bugs.

Apply test cases to mutant software

After testing the original software, apply the same test cases to the mutant software. If the output is different from the output in the original software, you will need to label the mutant as Dead. Continue to apply the other test cases to the mutant software and record the results. Dead mutants are no longer tested with the rest of the test set.

Compute mutation score

Mutation score is the ratio of the number of Dead Mutants over the number of Non Equivalent Mutants. The goal is to have a sore of one (1), which means that all faults in all mutants have been detected; the more dead mutants the higher the score will be.

Mutation Analysis of Result

The interpretation of the testing result is discussed below (Mutation Score Analysis)

Mutation Score Analysis

Using the mutation score formula, if you find that the test score is below one (1), then you should be able to analyze the data, because they are indicators of the status of the testing performed.

Following are ways to interpret the indicators –

The mutation testing result can be used as reference for the test case or test suite effectiveness. The mutation test score is directly proportional to the effectiveness of the test cases or test suite. If the score is close to or almost one, it is an indicator that the fault in a high percentage of mutants was detected. Equivalent Mutants produce similar outputs as the original program and so they cannot be dead. Take note that they are not included in the mutation score formula.

The live or equivalent mutants are indicators that the test cases are not adequate and hence need further improvements. Probably, the test did not test the code where the fault is. The tester should investigate and make a new test case if necessary. After improving on the test cases, the tester may repeat the procedure until a satisfactory mutation score is attained. The testing group management defines a satisfactory mutation score.

The mutation score could also be a good reference point on testing the effectiveness of the measures employed to improve the testing cases or suite. Mutation testing does not pass or fail the program under test. It just gives indicators to guide the testers on the improvements that need to be implementation.

Mutation testing will empower your software testing process with trust and fidelity. This will eventually help you to create a software application that can pass all other test procedures.

Mutation Testing Advantages and Disadvantages

Mutation testing helps tester to assess the quality of a test suite. This is possible by mutating certain elements of software in your source code and later checking and detecting if the test code is able to find the invisible errors. However, mutation testing is very costly to run, especially on very large and voluminous software applications.

Mutation testing is also a powerful tool to detect testing inadequacies or to check coverage on testing software. Software testers have known this method for many years. However, not many of them are using it for various reasons. There are several reasons that impede software industry from using this testing. However, this testing has its own share of advantages and disadvantages. Few of them are listed here below:

Advantages

• A powerful tool to determine the coverage of testing programs.
• Many steps used in this software are automated, like the creation of mutant software and the white box method for unit testing.
• It is capable of comprehensive testing of correctly chosen mutant programs.

Disadvantages

• You can create any number of mutants and the testing of these mutants is always costly and time consuming.
• It requires many test cases to distinguish the mutant from the original software.
• It also requires a lot of testing before a dependable data is obtained.
• It needs an automated tool to reduce testing time.
• It is not an applicable method for black box testing.
• It is very complicated to use without an automated tool.

The list of disadvantages mentioned above are the reasons why mutation testing has not really spread its usability outside of research. The following are what software developers want in relation to the future of mutation testing.

Automation of Mutation Testing

As mentioned earlier, mutation is complicated and requires numerous testing to have the needed data. Unlike in research where mutation testing has been used for decades, there are stiff delivery dates and compelling cost targets to meet.

Requirement for more researchers and users of the testing process is always a predominant factor. More numbers of users means there will be higher number of new developments and observations which can improve the usability of the program.

To get reliable results, you need to find better methods that can find equivalent mutants and reduce the number of tests. More equivalent numbers of mutants means longer testing time duration. 

Conclusion

Mutation testing is a misnomer. It is not just a testing method; it is more of an analytical method. Based on the set guidelines, the tester analyzes the results and implements the appropriate corrective measures.

Unlike the other testing methods, there is no Pass/ Fail disposition if the output fails to meet the standard output as defined in the test cases. Instead, adjustments are made to improve whatever was lacking or not up to standard, then test again until a satisfactory score is attained.

In real life, a 100% average testing effectiveness is not attainable, 100% can be achieved for a day or two, but not for several days, and certainly not on the average for weeks or months.

It is also an effective program for improving the quality of testing software but its widespread use is prevented by the difficulties encountered in using them. It would be very helpful to the developers of complex software if the testing methods could be improved to a point that it is not only effective in fault detection but also efficient as well.