A software testing department should never say that a piece of software is bug free. The most they can say is that there are no known defects.  Testing can show the presence of defects in a system; it cannot prove there are no remaining defects.

So how much testing is enough?

There are 3 tools that can be used in Software Testing to ensure test coverage:

  1. Test Strategy / Test Plan
  2. Test Case Requirement Matrix
  3. Test Case Code Coverage Matrix

Ensuring adequate test case coverage will prove that all critical business processes can be performed without faults and that any possible remaining defects will be likely be found in uncommon scenarios.

1. The SQA Test Plan

The challenge of the test plan is to ensure the test coverage is sufficient to prove the application is ready for release.

The test plan should demonstrate the test strategy to ensure:

  • that all critical business processes have been evaluated and are being tested
  • that the testing is realistic and reflects typical business scenarios
  • that all business risks have been addressed
  • that all compliance criteria has been met
  • that the necessary manpower and hardware resources are available
  • that all redundant testing has been eliminated

The test plan should ensure that by the completion of testing, management will have all the information necessary to reach a Go / No Go decision for launch.

2.  Test Case Requirement Matrix

The purpose of the test case matrix is to map one or more than one test case to each system requirement. The fundamental premise is that if one or more than one test case has been mapped to each requirement, then all the requirements of the system must have been tested and therefore the test case matrix proves testing is complete.

Some issues to be aware of when using the trace matrix:

  • If the requirements are not complete or not detailed enough then the test cases might fulfill the basic requirements but the testing won’t have provided what was needed.
  • If the requirements are clear but the test cases are insufficient (i.e. incomplete end state verification steps in the test case)  then a completed trace matrix still doesn’t indicate the testing coverage is complete.  You need to validate the quality of the test cases as well as the quality of the matrix.

Many defects can be found outside of the business and system requirements. Testing only according to specified requirements may be too narrowly focused to be 100% effective.

3. Test Case Code Coverage Matrix

Test case code coverage matrixes demonstrates the degree to which the source code of the program has been tested.  It is a form of testing that looks at the code directly and is often described as white box testing.  It requires that either the tester has some code development training or that a developer works closely with the tester to verify the matrix. 

To measure how well the program is exercised by a test suite,  one or more coverage criteria are used. There are a number of coverage criteria, the main ones are:

  • Function coverage – Has each function in the program been executed?
  • Statement coverage – Has each line of the source code been executed?
  • Condition coverage – Has each evaluation point (such as a true/false decision) been executed?
  • Path coverage – Has every possible route through a given part of the code been executed?
  • Entry/exit coverage – Has every possible call and return of the function been executed?