ISTQB CTFL Test Planning: Test Plans, Entry/Exit Criteria, Estimation and Prioritization

A test plan is a document that describes the objectives, resources and processes for a test project: it records the means and schedule for achieving the test objectives. It is a forward-looking document — it says how and when testing will be done, not what testing has already happened.

A test plan serves three purposes:

• Documents the means and schedule for achieving the test objectives.
• Helps ensure that the test activities performed will meet the established criteria.
• Communicates with team members and other stakeholders.

It also demonstrates that testing will adhere to the organization's existing test policy and test strategy — or explains why it deviates from them. The plan works within the test strategy; it does not replace it.

The planning activity has value beyond the document it produces. Planning guides testers' thinking and forces them to confront future challenges related to risks, schedules, people, tools, costs and effort before those challenges arrive. Planning is also not a one-off: the plan is refined as the project progresses and feedback comes in from monitoring and control.

Progress and completion reporting are covered in the Test Monitoring, Test Control and Test Completion lesson.

A typical test plan covers the context of testing, the people and risks involved, the test approach, and the budget and schedule. The CTFL syllabus lists this content:

• Context of testing — scope, test objectives, constraints and test basis.
• Assumptions and constraints of the test project.
• Stakeholders — roles, responsibilities, relevance to testing, and hiring and training needs.
• Communication — forms and frequency of communication, documentation templates.
• Risk register — product risks and project risks.
• Test approach — see below.
• Budget and schedule.

The test approach is the part most often confused with the others. It contains the:

• test levels and test types
• test techniques
• test deliverables
• entry criteria and exit criteria
• independence of testing
• metrics to be collected
• test data requirements and test environment requirements
• deviations from the organizational test policy and test strategy

What a test plan does not contain is the detail of individual tests — specific input values, expected results and step-by-step procedures are testware created later, during test design and implementation. A useful check: if an item describes how testing will be organized, it belongs in the plan; if it describes exactly what one test does, it does not.

Scope and objectives sit in the context section, not the approach; risks themselves sit in the risk register (how risks are identified and assessed is taught in the Risk Management lesson).

In iterative SDLCs there are two kinds of planning — release planning and iteration planning — and testers contribute to both, but at different levels of detail.

Release planning looks ahead to the release of a product, defines and re-defines the product backlog, and may involve refining larger user stories into smaller ones. It also serves as the basis for the test approach and test plan across all iterations. Testers involved in release planning:

• participate in writing testable user stories and acceptance criteria
• participate in project and quality risk analyses
• estimate test effort associated with user stories
• determine the test approach
• plan the testing for the release

Iteration planning looks ahead to the end of a single iteration and is concerned with the iteration backlog. Testers involved in iteration planning:

• participate in the detailed risk analysis of user stories
• determine the testability of user stories
• break down user stories into tasks (particularly testing tasks)
• estimate test effort for all testing tasks
• identify and refine functional and non-functional aspects of the test object

Entry criteria define the preconditions for undertaking an activity; exit criteria define what must be achieved to declare an activity completed. In short: entry answers "can we start?", exit answers "are we done?". Both are part of the test approach, are defined for each test level, and differ according to the test objectives.

A quick classification rule: a criterion that is about having what you need (environments, data, tools, a stable build that passed smoke tests, a baselined test basis) is an entry criterion. A criterion that is about results achieved (coverage reached, defects resolved, defect density below a threshold, regression tests automated) is an exit criterion.

Starting an activity without meeting its entry criteria does not save time: the activity is likely to become more difficult, time-consuming, costly and risky.

Can testing stop before the exit criteria are met?

Yes — running out of time or budget can be a valid exit criterion, but only if the stakeholders have reviewed and accepted the risk of going live without further testing. That decision is what makes the early stop legitimate.

The key point is who decides: accepting the residual risk of stopping early is a stakeholder decision, taken openly with the facts about what has and has not been tested. The exit criteria themselves stay as planned; the stakeholders' informed acceptance of the risk is what allows the activity to end.

In Agile software development, exit criteria are often called the Definition of Done, which defines the team's objective metrics for a releasable item. Entry criteria that a user story must fulfil to start development and/or testing are called the Definition of Ready. Acceptance criteria are different again: they describe the conditions a particular story's implementation must satisfy to be accepted, rather than the team-wide start or finish conditions.

Test effort estimation predicts the amount of test-related work needed to meet the objectives of a test project. Estimates are always based on assumptions and are subject to estimation error; estimating small tasks is usually more accurate than estimating large ones, so a large task can be decomposed into smaller tasks that are estimated individually. CTFL covers four techniques, two metrics-based and two expert-based.

Ratios work best when they come from the organization's own historical data for similar projects — that is generally the best source, because it reflects the organization's own processes.

Planning Poker, common in Agile teams, is a Wideband Delphi variant: each team member privately picks a numbered card, the cards are revealed together, large differences are discussed, and the team estimates again until it converges. The defining pattern is independent estimates, discussion of outliers, and repeated rounds.

Each technique reduces to simple arithmetic once you know what data it uses.

Estimation based on ratios

If past projects had a development-to-test effort ratio of D:T, then test effort = development effort × T / D. Example: a ratio of 4:1 and development effort of 600 person-days give 600 × 1 / 4 = 150 person-days of test effort. Check the direction of the ratio: test effort is the smaller share here, so the result must be smaller than the development effort.

Extrapolation

Measure the current project and apply the model the team has chosen. If the last three iterations took 30, 36 and 33 person-hours and the model is "average of the last three", the next iteration is estimated at (30 + 36 + 33) / 3 = 33 person-hours. Always apply the model the stem or the team states.

Three-point estimation

Experts give three estimates: the most optimistic (a), the most likely (m) and the most pessimistic (b). In its most popular form, the final estimate E is their weighted arithmetic mean, with the most likely value weighted four times:

E  = (a + 4m + b) / 6
SD = (b - a) / 6
result = E ± SD

Example: a = 6, m = 9, b = 24 person-days. E = (6 + 36 + 24) / 6 = 66 / 6 = 11. SD = (24 − 6) / 6 = 3. The result is 11 ± 3 person-days, i.e. between 8 and 14 person-days.

Remember that both values use a divisor of six, and that the answer is the estimate together with its measurement error: E is the centre of the range and SD its half-width.

Test case prioritization decides the order in which test cases are executed so that the most important tests run first. CTFL names three common strategies:

Recognizing the strategy from a description is mostly a matter of finding what the order is derived from. Ordering by likelihood and impact is risk-based, because those are the two factors of a risk level (how risk levels are assessed is covered in the Risk Management lesson). Ordering by stakeholder-assigned priorities of requirements or user stories, with tests traced to them, is requirements-based. Ordering by coverage figures is coverage-based.

Additional coverage prioritization

A variant of coverage-based prioritization: the test with the highest coverage runs first, and each subsequent test is the one that adds the most coverage not yet achieved. It is not the same as ordering by total coverage.

Worked example — statements 1–8. T1 covers 1–5; T2 covers 1, 2, 6; T3 covers 6, 7, 8; T4 covers 2, 3, 4, 8.

1. First: T1 (5 statements, the highest).
2. Uncovered now: 6, 7, 8. T2 adds 1 (6), T3 adds 3 (6, 7, 8), T4 adds 1 (8). Next: T3, even though T4 has the larger total.
3. Everything is covered; T2 and T4 add nothing, so they follow in any order (ties are broken by another rule, such as total coverage).

Ordering by total coverage instead would put T4 (4 statements) second although it adds only one new statement — a different and less efficient sequence.

Priority sets the default execution order, but it is not absolute: if a higher-priority test case depends on a lower-priority one, the lower-priority test case must be executed first, and the order must also take into account the availability of resources such as tools, environments or people that may only be available in a specific time window.

Applying this is a two-step process:

1. Place the constrained tests. A test that can only run in a particular slot (because its lab, device or specialist is only available then) goes in that slot, whatever its priority. A test that a higher-priority test depends on (for example, one that creates the data or state the other needs) goes before it.
2. Fill the rest by priority. All remaining tests follow their priority order.

Example (priority 1 = highest), five tests in five slots. R-1 has priority 1 and needs test data created by R-3 (priority 3). R-2 has priority 2 and R-4 has priority 4. R-5 has priority 5 but needs a security specialist who is available only in the third slot. Step 1 fixes R-5 in slot 3 and puts R-3 before R-1. Step 2 fills the free slots by priority, respecting that dependency: R-3 (slot 1), R-1 (slot 2), R-5 (slot 3), R-2 (slot 4), R-4 (slot 5). Pure priority order (R-1, R-2, R-3, R-4, R-5) fails twice: R-1 would run before its test data exists, and R-5 would miss the specialist's slot.

The test pyramid is a model showing that different tests may have different granularity: the lower the layer, the smaller, more isolated and faster the tests, and the more of them are needed; the higher the layer, the more complex, end-to-end and slower the tests, and the fewer of them are needed. It supports the team in test automation and in test effort allocation.

The number and naming of the layers vary. The original model has three layers — unit tests, service tests and UI tests; another popular model uses unit (component) tests, integration (component integration) tests and end-to-end tests. Other test levels can also be used.

The pyramid is a guide to where automation effort should go. A suite dominated by slow end-to-end UI tests with few unit tests is effectively inverted; the pyramid points towards growing the base of component tests and keeping the end-to-end layer thin.

The testing quadrants, defined by Brian Marick and used in Agile software development, group test levels and test types along two dimensions so that all appropriate types of testing are considered and the differences between them can be explained to stakeholders. In this model tests can be business-facing or technology-facing, and they can support the team (guide development) or critique the product (measure its behaviour against expectations).

The point most easily confused is usability. Usability is a non-functional quality characteristic, yet usability testing sits in Q3, with exploratory and user acceptance testing, because it is user-oriented and critiques the product from the business side. All other non-functional testing, such as performance and security testing, sits in Q4.

A practical way to place a test is to ask two questions: is it expressed in terms the business understands, or in technical terms? And does it help the team build the product right as it goes, or does it evaluate the finished behaviour? The test levels and test types themselves are explained in the test levels and test types lesson of the SDLC domain.