ISTQB CTFL Test Monitoring, Test Control and Test Completion: Metrics, Test Reports and Status Communication

They are three related management activities that run alongside the rest of the test process. Test monitoring gathers information about testing; test control acts on that information; test completion consolidates the results when a stage of testing ends. The general test process (which these activities belong to) is described in the test activities lesson; this lesson covers the management detail the syllabus adds in section 5.3.

A simple way to separate them: if someone is collecting or comparing data, it is monitoring; if someone is changing how testing proceeds, it is control; if someone is closing off a finished stage, it is completion. Test planning comes before all three: it sets the exit criteria that monitoring measures against.

Test monitoring is concerned with gathering information about testing. That information is used to assess test progress and to measure whether the exit criteria, or the test tasks associated with them, are satisfied. A typical example is checking whether the planned coverage of product risks, requirements or acceptance criteria has been reached.

Monitoring itself does not change the plan. Reading a burn-down chart, or checking how many of the identified product risks have been covered by tests so far, is monitoring, even when what it shows is bad news. Doing something about the problem is the next activity, test control.

Test control uses the information from test monitoring to give guidance and the corrective actions needed to make testing as effective and efficient as possible. These actions are called control directives. Each one fits a particular kind of deviation:

All four are genuine control directives; what decides between them is the cause of the deviation. A change in risk calls for a change in the order of tests, rework calls for re-checking the criteria, a late dependency calls for a schedule change, and a shortfall of capacity calls for resources. Control means acting while testing is still running: noting a problem without changing anything is still only monitoring.

Test completion collects data from completed test activities to consolidate experience, testware and any other relevant information. It takes place at project milestones, such as when:

• a test level is completed;
• an agile iteration is finished;
• a test project is completed or cancelled;
• a software system is released;
• a maintenance release is completed.

Two consequences are worth remembering. First, the list includes a cancelled project, so stopping early does not remove the need to close testing down properly. Second, the milestone is the trigger, not the exit criteria. Completion is ideally reached with the exit criteria met; when they are not, completion still takes place and the completion report states which criteria were not achieved and what risk and defects remain. Day-to-day reporting while testing is still running belongs to monitoring, not completion.

Test metrics are collected to show three things:

• progress against the planned schedule and budget;
• the current quality of the test object;
• the effectiveness of the test activities with respect to the objectives or an iteration goal.

Test monitoring gathers a variety of metrics to support test control and test completion.

The syllabus groups common test metrics into seven categories. Classify a metric by asking what it describes: the project, the testing, the product, the defects, the risk, the coverage or the cost.

Near-twins to keep apart

• Test effort vs test execution time. Test effort (how much work the project has spent on testing) is a project progress metric, next to task completion and resource usage. Test execution time (how long the tests take to run) is a test progress metric.
• Defect detection percentage vs defect density. Defect detection percentage compares the defects found by testing with all defects found, including those found later. For example, if testing found 120 defects and another 30 were reported after release, testing found 120 of 150, which is 80%. Defect density relates the number of defects to the size of the test object, for example defects per thousand lines of code.
• Product quality vs defect metrics. Mean time to failure, availability and response time describe how the system behaves, so they are product quality metrics even though failures are involved; defect metrics count or relate defects.
• Requirements coverage vs code coverage. Both are coverage metrics. Requirements coverage relates tests to the test basis (how many requirements have tests traced to them); code coverage relates tests to the implementation (how much of the code the tests execute).
• Cost of testing vs organizational cost of quality. Both are cost metrics. Cost of testing is limited to the test activities; organizational cost of quality is a wider, organization-level view of what quality costs. Both are expressed in money, which separates them from the effort and resource-usage figures of project progress metrics.

A test progress report supports the ongoing control of testing. It must give enough information to modify the test schedule, resources or test plan when testing deviates from the plan or circumstances change. Progress reports are produced regularly while testing runs, for example daily or weekly.

Typical test progress reports include:

• the testing period;
• test progress (for example ahead of or behind schedule), including any notable deviations;
• impediments to testing and their workarounds;
• test metrics (see the categories above);
• new and changed risks within the testing period;
• testing planned for the next period.

Notice the direction of this list: it describes the period just ended and looks ahead to the next one, because its reader needs to decide what to change now.

A test completion report summarizes a specific stage of testing, such as a test level, test cycle or iteration, and can give information for subsequent testing. It is prepared during test completion, when a project, test level or test type is complete and, ideally, its exit criteria have been met. It draws on the test progress reports and other data from that stage.

Typical test completion reports include:

• a test summary;
• an evaluation of testing and product quality based on the original test plan (its test objectives and exit criteria);
• deviations from the test plan (for example differences from the planned schedule, duration and effort);
• testing impediments and workarounds;
• test metrics based on the progress reports;
• unmitigated risks and defects not fixed;
• lessons learned that are relevant to the testing.

Progress report vs completion report

Different audiences need different information in test reports, and the audience also influences the degree of formality and the frequency of reporting. Reporting on progress within a team may be frequent and informal, while reporting on a completed project follows a set template and happens only once. For example, a test lead may give the developers a quick, detail-heavy update several times a week, while the steering committee receives a structured monthly summary.

The ISO/IEC/IEEE 29119-3 standard (the test documentation part of the 29119 series) includes templates and examples for test progress reports, which it calls test status reports, and for test completion reports. Note the naming difference: the syllabus says “test progress report” where the standard says “test status report”.

The best means of communicating test status depends on the stakeholders and the situation. The syllabus lists these options:

• verbal communication with team members and other stakeholders;
• dashboards, such as CI/CD dashboards, task boards and burn-down charts;
• electronic communication channels, such as email and chat;
• online documentation;
• formal test reports.

One or more of these may be used. Which ones are chosen depends on test management concerns, the organizational test strategy, regulatory standards or, for self-organizing teams, the team itself.

Two further rules guide the choice:

• More formal communication may suit distributed teams. When direct face-to-face communication is not always possible because of geographical distance or time differences, status needs a form that people can read whenever and wherever they are, rather than one that only works for those present at the time.
• Tailor the communication to each stakeholder. Different stakeholders are interested in different information. A tester fixing tests needs the detail of which test failed and why; a product manager deciding on a launch date needs the overall picture of progress, quality and remaining risk. Matching the content and level of detail to each reader is the point.