CTFL Why Testing Is Necessary: QA vs QC, Errors and Failures

Testing is necessary because it is a cost-effective way to find defects, a direct measure of quality, a stand-in for users who cannot be involved throughout development, and sometimes a contractual, legal or regulatory obligation. As a form of quality control, it helps a project reach its agreed goals within the set scope, time, quality and budget constraints.

The syllabus names four distinct contributions. Each one answers a different question, so a scenario usually points at exactly one of them:

Contribution to success is not limited to the test team: any stakeholder who applies testing skills, such as a business analyst reviewing a requirement, helps the project too.

Testing improves the test object indirectly: it detects defects, and the improvement happens when those defects are removed by debugging, which is a separate, non-testing activity. A tester who reports a failure has made the fix possible but has not changed the product; the developer who locates and corrects the defect is the one who changes it.

Two wrong ideas follow from blurring this line:

• "Testers remove defects during test execution." Locating and correcting the cause is debugging. (The full testing-versus-debugging distinction belongs to the lesson on what testing is.)
• "Testing showed the product has no defects left." Testing can show defects are present but cannot prove there are none, so it never certifies a defect-free product.

Testing also covers more than code: testing components, systems and their documentation all help identify defects.

Testing provides a direct evaluation of the test object's quality at various stages of the SDLC, and those measures feed project management decisions such as whether to move to the next stage or whether to release.

This contribution is about information, not repair. When a test manager reports coverage and pass rates at the end of integration testing and the project decides the build is ready to enter system testing, nothing in the product has been fixed by that report; testing has measured the product and the measurement has informed a decision. That makes it different from defect detection (which leads to a fix) and from quality assurance (which would change a process).

Testing gives users indirect representation on a development project: testers make sure their understanding of users' needs is considered throughout the development lifecycle. The alternative, involving a representative set of real users all the way through, is not usually possible because of the high cost and the lack of suitable, available users. That cost and availability problem is the reason testers act on users' behalf, not a claim that testers are more skilled or more objective than users.

Testing may also be required to meet contractual or legal requirements, or to comply with regulatory standards. A customer contract may specify test levels or evidence, and an aviation or rail safety authority may refuse certification of a safety-critical system without documented testing. In these cases the reason for testing comes from outside the project.

Testing and quality assurance (QA) are not the same thing: testing is a form of quality control (QC), and QA is about processes. People often use "testing" and "QA" interchangeably, but the syllabus separates them sharply.

QA applies to both development and testing processes, and it is the responsibility of everyone on a project, not of a separate QA team.

The pairing matters because the wrong combinations are easy to mix up: QA is process + preventive, QC is product + corrective. "Process-oriented, corrective" or "product-oriented, preventive" describe neither.

The same test results serve both: QC uses them to get defects fixed, and QA uses them as feedback on how well the development and test processes are working.

This gives a practical way to tell which one is happening, or which one is missing:

• When results are used to correct the product (a reported defect is fixed and the fix is confirmed), they are serving QC.
• When results are used to change how the team works (for example, defect data shows many problems start in unclear interface agreements, so the team adds an interface design walkthrough), they are serving QA as process feedback.

A product can be well controlled while its process stays unimproved. Fixing everything that testing finds says something about this product; it says nothing about whether the process that keeps producing those defects is getting better. That second question is the one QA asks.

An error is a human action that produces an incorrect result; a defect is an imperfection or deficiency in a work product where it does not meet its requirements or specifications; a failure is an event in which a component or system does not perform a required function within specified limits. People make errors (mistakes), which produce defects (also called faults or bugs), which in turn may result in failures.

To classify any item in a scenario, ask what it is: something a person did (error), something wrong inside a work product (defect), something the running system did or failed to do (failure), or the circumstance that made the person go wrong (root cause). A wrong line of code is never a failure by itself; it is a defect that may or may not fail.

Defects can be in any work product, not only code, and a defect does not always cause a failure.

Defects are found in documentation (a requirements specification, a test script), in source code, and in supporting work products such as a build file. Defects in work products produced earlier in the SDLC, if they go undetected, often lead to defective work products later: an incorrect formula in a design document can be coded faithfully, so one design defect becomes a code defect too. This is one reason early testing and reviews pay off.

When defective code is executed, the system may fail to do something it should, or do something it should not. But execution does not guarantee a failure:

• some defects cause a failure every time they are executed;
• some cause a failure only in specific circumstances (for example, only on 29 February, or only when a particular time zone is set);
• some may never cause a failure in the whole life of the system.

So a defect whose triggering condition has never occurred is still a defect, even though no failure has happened yet. And a clean test run does not rule out failures in operation, because the conditions that trigger a defect may simply not have been exercised.

Errors and defects are not the only causes of failures: environmental conditions can cause them too, for example when radiation or electromagnetic fields cause defects in firmware.

The chain still runs through a defect. If radiation in orbit corrupts some bits of a satellite controller's correctly written firmware, the corrupted firmware now contains a defect, and running it produces the failure (the controller restarts unexpectedly, say). What is different is the origin: no human error introduced that defect. This is the exception to "every defect starts with a mistake".

A root cause is a fundamental reason for the occurrence of a problem, such as a situation that leads to an error; root cause analysis identifies it so that similar failures or defects can be prevented or made less frequent.

People make errors for many reasons: time pressure; complexity of work products, processes, infrastructure or interactions; tiredness; or a lack of adequate training. Those situations are typical root causes. Root cause analysis is typically performed when a failure occurs or a defect is identified.

The key distinction is between fixing this problem and preventing the next one:

Detecting and correcting defects still matters, but only an action on the underlying situation prevents similar defects or makes them less frequent.