Logo

Navigation
  • Home
  • Services
    • ERP Solutions
    • Implementation Solutions
    • Support and Maintenance Solutions
    • Custom Solutions
    • Upgrade Solutions
    • Training and Mentoring
    • Web Solutions
    • Production Support
    • Architecture Designing
    • Independent Validation and Testing Services
    • Infrastructure Management
  • Expertise
    • Microsoft Development Expertise
    • Mobile Development
    • SQL Server Database and BI
    • SAP BI, SAP Hana, SAP BO
    • Oracle and BI
    • Oracle RAC
  • Technical Training
    • Learn Data Management
      • Business Intelligence
      • Data Mining
      • Data Modeling
      • Data Warehousing
      • Disaster Recovery
    • Learn Concepts
      • Application Development
      • Client Server
      • Cloud Computing Tutorials
      • Cluster Computing
      • CRM Tutorial
      • EDI Tutorials
      • ERP Tutorials
      • NLP
      • OOPS
      • Concepts
      • SOA Tutorial
      • Supply Chain
      • Technology Trends
      • UML
      • Virtualization
      • Web 2.0
    • Learn Java
      • JavaScript Tutorial
      • JSP Tutorials
      • J2EE
    • Learn Microsoft
      • MSAS
      • ASP.NET
      • ASP.NET 2.0
      • C Sharp
      • MS Project Training
      • Silverlight
      • SQL Server 2005
      • VB.NET 2005
    • Learn Networking
      • Networking
      • Wireless
    • Learn Oracle
      • Oracle 10g
      • PL/SQL
      • Oracle 11g Tutorials
      • Oracle 9i
      • Oracle Apps
    • Learn Programming
      • Ajax Tutorial
      • C Language
      • C++ Tutorials
      • CSS Tutorial
      • CSS3 Tutorial
      • JavaScript Tutorial
      • jQuery Tutorial
      • MainFrame
      • PHP Tutorial
      • VBScript Tutorial
      • XML Tutorial
    • Learn Software Testing
      • Software Testing Types
      • SQA
      • Testing
  • Career Training
    • Career Improvement
      • Career Articles
      • Certification Articles
      • Conflict Management
      • Core Skills
      • Decision Making
      • Entrepreneurship
      • Goal Setting
      • Life Skills
      • Performance Development
      • Personal Excellence
      • Personality Development
      • Problem Solving
      • Relationship Management
      • Self Confidence
      • Self Supervision
      • Social Networking
      • Strategic Planning
      • Time Management
    • Education Help
      • Career Tracks
      • Essay Writing
      • Internship Tips
      • Online Education
      • Scholarships
      • Student Loans
    • Managerial Skills
      • Business Communication
      • Business Networking
      • Facilitator Skills
      • Managing Change
      • Marketing Management
      • Meeting Management
      • Process Management
      • Project Management
      • Project Management Life Cycle
      • Project Management Process
      • Project Risk Management
      • Relationship Management
      • Task Management
      • Team Building
      • Virtual Team Management
    • Essential Life Skills
      • Anger Management
      • Anxiety Management
      • Attitude Development
      • Coaching and Mentoring
      • Emotional Intelligence
      • Stress Management
      • Positive Thinking
    • Communication Skills
      • Conversation Skills
      • Cross Culture Competence
      • English Vocabulary
      • Listening Skills
      • Public Speaking Skills
      • Questioning Skills
    • Soft Skills
      • Assertive Skills
      • Influence Skills
      • Leadership Skills
      • Memory Skills
      • People Skills
      • Presentation Skills
    • Finding a Job
      • Etiquette Tips
      • Group Discussions
      • HR Interviews
      • Interview Notes
      • Job Search Tips
      • Resume Tips
      • Sample Resumes
 

What is Mutation Testing

By Exforsys | on December 21, 2011 |
Software Testing Types

As the demand for software grows, so is the complexity of the design. The more complex a software is, the higher will be the testing needs, quality assurance and customer satisfaction. Although testing is, an integral part of the software development process the issue is still on what is sufficient or adequate testing still open.

Topics

  • What is Mutation Testing?
  • Mutation Testing Procedure
  • Mutation Score Analysis
  • Mutation Testing Advantages
  • Mutation Testing Disadvantages
  • Automation of Mutation Testing
  • Conclusion

If your tests cannot find a bug, can you believe that there are no bugs? Is it possible to trust your unit tests? Can you ascertain that they are telling truth for you? How does it work out to assess your tests? Mutation test is one of those wonderful tests that allow you to assess the tests. Mutation test involves deliberately altering, modifying or changing a program code, later rerunning a suit of correct unit tests against the mutated program.

In nutshell, mutation test works in the following manner. Initially, you will need to start the test with a piece of production code that the unit tests cover in the right way. Once you ascertain that all tests pass for a given code, and then you can start applying mutation to the target assembly.

Testing means additional cost and less productivity. For optimum development processing, there is a great need for the continuous improvement of quality testing methods. Testing coverage is an important issue in software development. Testing coverage is how well you test the software. The question is when do you stop testing? What is “adequate” testing? Right now, the most common practice is to state the requirement of percentage coverage like “a minimum xyz% coverage”. You will need to balance various requirements to bring about quality and customer satisfaction against productivity and cost.

Another important question to consider is, “How extensive are the testing programs?” What is the assurance for the “xyz% coverage” claim? What determines that at least “xyz%” of all possible causes of errors, bugs or failures is covered by the existing testing methods? Mutation testing is one way to verify if the software tester performed the testing in a proper manner. Through mutation testing, it can be determined, if the set of testing methods used in the developing process were appropriate and adequate to ensure product quality.

What is Mutation Testing?

To determine the correctness of a testing program, you will need to observe the behavior for each test case. If there are no detected faults, then the program is correct or it passes the test. A test case is the set of input values to the program under test and the corresponding output values.

Mutation testing is a method or strategy to check the effectiveness or accuracy of a testing program to detect the fault in the system or program under test. The test is called so because mutants of the software are created and run with the test cases.

Experts also call mutation testing as fault-based testing strategy because the mutants are created by introducing a single fault into the original program. You can set any number of mutants into the system. There are other versions of mutation testing, but they rely on the same method – to mutate the original software.

Some of the other versions are weak mutation, interface mutation and specification based mutation. Mutation testing is not a new method, developers know about it since the late 1970s. However, researchers have been using it in the educational institutions than in the industrial software domain.

Mutation Testing Procedure

The following steps rely on the traditional mutation process:

Create mutant software

The system /program under test are modified by rewriting the source code to introduce a single fault. The new versions of the system/program are mutants of the original ones. Each mutant has one fault, different from the original software and the other mutants. The tester may create as many mutants as needed. There is no limit.

Prepare test cases for input to the original and mutant software

The test cases should plan for the detection of the fault introduced into the mutant software. The tester may improve on the test cases, if necessary, so that the fault in the mutant software is found.

Apply test cases to original software

In the original software, the output is supposed to be correct, or must be similar as the expected output. If not, then there would be an error in the application, which must be fixed. Apply the test cases again until there are no bugs.

Apply test cases to mutant software

After testing the original software, apply the same test cases to the mutant software. If the output is different from the output in the original software, you will need to label the mutant as Dead. Continue to apply the other test cases to the mutant software and record the results. Dead mutants are no longer tested with the rest of the test set.

Compute mutation score

Mutation score is the ratio of the number of Dead Mutants over the number of Non Equivalent Mutants. The goal is to have a sore of one (1), which means that all faults in all mutants have been detected; the more dead mutants the higher the score will be.

Mutation Analysis of Result

The interpretation of the testing result is discussed below (Mutation Score Analysis)

Mutation Score Analysis

Using the mutation score formula, if you find that the test score is below one (1), then you should be able to analyze the data, because they are indicators of the status of the testing performed.

Following are ways to interpret the indicators –

The mutation testing result can be used as reference for the test case or test suite effectiveness. The mutation test score is directly proportional to the effectiveness of the test cases or test suite. If the score is close to or almost one, it is an indicator that the fault in a high percentage of mutants was detected. Equivalent Mutants produce similar outputs as the original program and so they cannot be dead. Take note that they are not included in the mutation score formula.

The live or equivalent mutants are indicators that the test cases are not adequate and hence need further improvements. Probably, the test did not test the code where the fault is. The tester should investigate and make a new test case if necessary. After improving on the test cases, the tester may repeat the procedure until a satisfactory mutation score is attained. The testing group management defines a satisfactory mutation score.

The mutation score could also be a good reference point on testing the effectiveness of the measures employed to improve the testing cases or suite. Mutation testing does not pass or fail the program under test. It just gives indicators to guide the testers on the improvements that need to be implementation.

Mutation testing will empower your software testing process with trust and fidelity. This will eventually help you to create a software application that can pass all other test procedures.

Mutation Testing Advantages and Disadvantages

Mutation testing helps tester to assess the quality of a test suite. This is possible by mutating certain elements of software in your source code and later checking and detecting if the test code is able to find the invisible errors. However, mutation testing is very costly to run, especially on very large and voluminous software applications.

Mutation testing is also a powerful tool to detect testing inadequacies or to check coverage on testing software. Software testers have known this method for many years. However, not many of them are using it for various reasons. There are several reasons that impede software industry from using this testing. However, this testing has its own share of advantages and disadvantages. Few of them are listed here below:

Advantages

• A powerful tool to determine the coverage of testing programs.
• Many steps used in this software are automated, like the creation of mutant software and the white box method for unit testing.
• It is capable of comprehensive testing of correctly chosen mutant programs.

Disadvantages

• You can create any number of mutants and the testing of these mutants is always costly and time consuming.
• It requires many test cases to distinguish the mutant from the original software.
• It also requires a lot of testing before a dependable data is obtained.
• It needs an automated tool to reduce testing time.
• It is not an applicable method for black box testing.
• It is very complicated to use without an automated tool.

The list of disadvantages mentioned above are the reasons why mutation testing has not really spread its usability outside of research. The following are what software developers want in relation to the future of mutation testing.

Automation of Mutation Testing

As mentioned earlier, mutation is complicated and requires numerous testing to have the needed data. Unlike in research where mutation testing has been used for decades, there are stiff delivery dates and compelling cost targets to meet.

Requirement for more researchers and users of the testing process is always a predominant factor. More numbers of users means there will be higher number of new developments and observations which can improve the usability of the program.

To get reliable results, you need to find better methods that can find equivalent mutants and reduce the number of tests. More equivalent numbers of mutants means longer testing time duration. 

Conclusion

Mutation testing is a misnomer. It is not just a testing method; it is more of an analytical method. Based on the set guidelines, the tester analyzes the results and implements the appropriate corrective measures.

Unlike the other testing methods, there is no Pass/ Fail disposition if the output fails to meet the standard output as defined in the test cases. Instead, adjustments are made to improve whatever was lacking or not up to standard, then test again until a satisfactory score is attained.

In real life, a 100% average testing effectiveness is not attainable, 100% can be achieved for a day or two, but not for several days, and certainly not on the average for weeks or months.

It is also an effective program for improving the quality of testing software but its widespread use is prevented by the difficulties encountered in using them. It would be very helpful to the developers of complex software if the testing methods could be improved to a point that it is not only effective in fault detection but also efficient as well.

« « What is Monkey Testing
What is Negative Testing » »

Author Description

Avatar

Editorial Team at Exforsys is a team of IT Consulting and Training team led by Chandra Vennapoosa.

Free Training

RSSSubscribe 391 Followers
  • Popular
  • Recent
  • What is Acceptance Testing

    May 10, 2011 - 0 Comment
  • What is Sanity Testing

    January 23, 2012 - 0 Comment
  • What is End-to-End Testing

    August 2, 2011 - 0 Comment
  • What is Security Testing

    January 27, 2012 - 0 Comment
  • What is Beta Testing

    June 1, 2011 - 0 Comment
  • What is Dynamic Testing

    July 29, 2011 - 0 Comment
  • What is Smoke Testing

    February 6, 2012 - 0 Comment
  • What is Concurrent Testing

    June 15, 2011 - 0 Comment
  • What is Functional Testing

    December 5, 2011 - 0 Comment
  • What is Static Testing

    February 10, 2012 - 0 Comment
  • What is White Box Testing

    February 22, 2012 - 0 Comment
  • What is Usability Testing

    February 16, 2012 - 0 Comment
  • What is Stress Testing

    February 15, 2012 - 0 Comment
  • What is Static Testing

    February 10, 2012 - 0 Comment
  • What is Smoke Testing

    February 6, 2012 - 0 Comment
  • What is Security Testing

    January 27, 2012 - 0 Comment
  • What is Sanity Testing

    January 23, 2012 - 0 Comment
  • What is Regression Testing

    January 11, 2012 - 0 Comment
  • What is Data Recovery Testing

    January 9, 2012 - 0 Comment
  • What is Performance Testing

    January 1, 2012 - 0 Comment

Exforsys e-Newsletter

ebook
 

Related Articles

  • What is White Box Testing
  • What is Usability Testing
  • What is Stress Testing
  • What is Static Testing
  • What is Smoke Testing

Latest Articles

  • Project Management Techniques
  • Product Development Best Practices
  • Importance of Quality Data Management
  • How to Maximize Quality Assurance
  • Utilizing Effective Quality Assurance Strategies
  • Sitemap
  • Privacy Policy
  • DMCA
  • Trademark Information
  • Contact Us
© 2023. All Rights Reserved.IT Training and Consulting
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.AcceptReject Read More
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT