ISTQB CTFL Feedback and Review Process: Activities, Roles, Review Types and Success Factors

Early and frequent stakeholder feedback lets potential quality problems be communicated early, while they are still cheap to fix. It is the people side of static testing: instead of waiting to show stakeholders a finished product, the team asks them throughout development whether what is being built is what they need.

The CTFL syllabus lists these benefits of frequent feedback:

• Potential quality problems are communicated early, before they turn into expensive rework.
• Misunderstandings about requirements are prevented, because stakeholders see and correct the team's interpretation.
• Changes to requirements are understood and implemented sooner, rather than discovered at the end.
• The development team gains a better understanding of what it is building.
• The team can focus on the features that bring the most value to stakeholders and have the most positive impact on the identified risks.

Frequent feedback improves the team's understanding of the requirements so that the right product is built and tested.

What happens when stakeholders are not involved

When stakeholders are involved only at the start (or hardly at all), the product may not meet their original or current vision: their needs move on while the team builds to an old specification. Failing to deliver what stakeholders want can lead to costly rework, missed deadlines, blame games and even complete project failure.

The CTFL review process has five activities, based on the generic process in the ISO/IEC 20246 standard: planning, review initiation, individual review, communication and analysis, and fixing and reporting. The process is a flexible framework: the more formal the review, the more of the tasks within each activity are performed. A large work product is often too big for a single review, so the process may run several times to cover all of it.

A useful way to place a task is to ask who is acting and on what: the scope and exit criteria are set before anyone reads (planning); access, roles and materials are checked before reading starts (initiation); one person reading alone and logging findings is individual review; the group deciding what each finding means is communication and analysis; turning confirmed defects into tracked reports and closing the review is fixing and reporting.

Planning decides what the review is for and when it is finished; review initiation makes sure the people can actually start. They are easy to confuse because both happen before anyone reads the work product.

Planning defines the scope of the review. That includes the purpose of the review, which work product (or which part of it) is reviewed, the quality characteristics to be evaluated, the areas to focus on, the exit criteria, supporting information such as standards, and the effort and timeframes. Decisions about individual anomalies, and defect reports, cannot be made here: no anomalies exist yet.

Review initiation checks readiness. Each participant must have access to the work product, understand their role and responsibilities, and receive everything they need to perform the review.

In individual review, each reviewer works alone to assess the quality of the work product and identify anomalies, recommendations and questions, applying one or more review techniques. The CTFL syllabus names checklist-based reviewing and scenario-based reviewing as examples. Reviewers log everything they find: anomalies, recommendations and questions.

Individual review does not decide anything. Agreeing whether a finding is a defect, who owns it and what must happen next belongs to the next activity. Keeping the two apart lets each reviewer concentrate on the work product itself; that concentration is also why reviews of long work products are split into smaller chunks (see the success factors below).

Communication and analysis exists because an anomaly is not necessarily a defect. A reviewer may have misunderstood the work product, or the finding may be a question or a suggestion. So every logged anomaly is analysed and discussed, and for each one a decision is made on its status, ownership and required actions. This typically happens in a review meeting, where participants also decide the quality level of the reviewed work product and the follow-up actions required. A follow-up review may be needed to complete those actions.

The point of the discussion is to separate real defects from misunderstandings before effort is spent on reports and fixes.

Fixing and reporting comes last. A defect report is created for every defect so that corrective actions can be followed up. The work product can be accepted only once the exit criteria are reached, and the review results are reported.

The CTFL syllabus names six principal review roles. One person can hold more than one role (subject to the rules of the review type), and the ISO/IEC 20246 standard describes further, more detailed roles.

Review leader, moderator or manager?

These three are the most often confused. The manager owns the resources: what gets reviewed and the staff and time spent on it. The review leader owns the review as a whole: who takes part, when and where. The moderator owns the meeting while it runs: mediating disagreements, keeping time and keeping the atmosphere safe.

Reviewers are not limited to testers, and independence is not what defines them: developers, business analysts, domain experts and users can all review. Recording decisions is the scribe's job, and deciding what is reviewed is the manager's.

CTFL compares four commonly used review types, from least to most formal: informal review, walkthrough, technical review and inspection. They differ in who leads, how much of the generic process they follow and what they are mainly for.

Telling them apart

• If the author presents the work product to teach the audience or gather ideas, it is a walkthrough.
• If experts meet under a moderator to settle a technical problem, it is a technical review.
• If the goal is as many anomalies as possible, with the full process and metrics, it is an inspection.
• If there is no defined process and no formal output, it is an informal review (a pair review or a quick colleague check, for example).

The inspection's role rule is a frequent trap: because the review must be independent of the author, the author may not act as review leader or as scribe. The author can still take part as author and fix the work product.

There is no single correct review type; the choice is tailored to the context. The level of formality depends on factors such as:

• the software development lifecycle model in use;
• the maturity of the development process;
• the criticality and complexity of the work product;
• legal or regulatory requirements;
• the need for an audit trail.

The choice of review type also depends on project needs, available resources, the product type and its risks, the business domain and the company culture.

Two consequences follow. First, one organisation will often use several review types side by side, matching each to the work product and its context. Second, the same work product can be reviewed more than once with different review types, for example an informal review first and a more formal one later.

The CTFL syllabus lists these factors that contribute to successful reviews:

• Clear objectives and measurable exit criteria. Evaluation of participants should never be an objective.
• Choosing the appropriate review type to achieve the objectives and to suit the type of work product, the participants, the project needs and the context.
• Conducting reviews on small chunks, so that reviewers do not lose concentration during individual review or the review meeting.
• Providing feedback from reviews to stakeholders and authors, so they can improve the product and their activities.
• Providing adequate time for participants to prepare.
• Support from management for the review process.
• Making reviews part of the organisation's culture, to promote learning and process improvement.
• Providing adequate training for all participants so they know how to fulfil their role.
• Facilitating meetings.