ISTQB CTFL Black-box Test Techniques: Equivalence Partitioning, Boundary Value Analysis, Decision Tables and State Transitions

Black-box test techniques derive test conditions, test cases and test data from the specified behaviour of the test object, without looking at its internal structure. The CTFL syllabus covers four of them, and each one models the requirements in a different way, so each finds a different kind of defect:

All four are K3 objectives: you are expected to apply them to a short specification and work out coverage items, a minimum number of test cases or a coverage percentage. Coverage is always calculated the same way: coverage items exercised divided by coverage items identified, as a percentage. How black-box techniques differ from white-box and experience-based techniques is covered in the test techniques overview lesson.

Equivalence partitioning divides data into partitions (equivalence classes) based on the expectation that all elements of a partition are processed the same way by the test object. If a test with one value from a partition detects a defect, tests with other values from that partition should detect it too, so one value represents the whole partition.

Rules for partitions

• Partitions can be identified for any data element related to the test object: inputs, outputs, configuration items, internal values, time-related values and interface parameters, not only input fields.
• A partition can be continuous or discrete, ordered or unordered, finite or infinite. A whole-number range is ordered; a list of payment methods or countries is unordered.
• Partitions must not overlap and must be non-empty sets: every value belongs to exactly one partition, and a partition with a single value is still a partition.
• A valid partition contains values the test object should process according to its specification; an invalid partition contains values it should ignore or reject. Exactly what counts as valid or invalid can vary between teams and organizations: valid may mean values the specification defines processing for, and invalid values for which no processing is defined. Either way, state the interpretation you use and apply it consistently.

Identifying the partitions

Read the specification and split the whole domain, including what is rejected. A field that accepts whole numbers 100 to 999 has three partitions: below 100 (invalid), 100 to 999 (valid) and above 999 (invalid). If the specification also defines a separate response for entries that are not whole numbers (text or decimals), that is one more, unordered invalid partition. If valid values are processed differently (for example 1 to 9 items shipped at the normal rate and 10 to 99 items with a bulk discount), each processing rule is its own valid partition.

Minimum tests and coverage

The coverage items are the partitions. 100% equivalence partition coverage requires test cases that exercise every identified partition, valid and invalid, at least once, so the minimum number of test values for a single input equals its number of partitions. Coverage is partitions exercised divided by partitions identified. Two test values in the same partition count once, and a value that merely looks like a boundary covers only the one partition it falls in. Always include the invalid partitions in both the numerator and the denominator.

Each Choice coverage is the simplest coverage criterion for a test object with several sets of partitions, for example one set per input or configuration parameter. It is achieved when each partition from each set is exercised by at least one test case; combinations of partitions are not taken into account.

Because one test case picks one partition from every set at the same time, a single test can cover a partition of each input. The minimum number of test cases is therefore the size of the largest set: shorter sets simply repeat a value. For three inputs with 2, 5 and 3 partitions, five tests are enough. Adding the counts ignores this sharing, and multiplying them would test every combination, which is a much stronger and more expensive goal.

When some tests already exist, list which partitions of each set are still missing. Each additional test can take one missing partition from every set, so the number of additional tests needed is the largest number of missing partitions in any one set. Duplicate existing tests add nothing.

Boundary value analysis (BVA) exercises the boundaries of equivalence partitions. It can only be used for ordered partitions: the minimum and maximum values of a partition are its boundary values, and an unordered category (seat class, country, payment method) has no smallest or largest value. A partition is ordered when its values have a natural order, so it has a smallest and a largest value.

BVA focuses on boundaries because developers are more likely to make mistakes there: a boundary may be misplaced in the specification or coded wrongly, such as < instead of <=. Every partition next to a boundary contributes a boundary value, including the invalid ones: for a valid range 200 to 500 of whole numbers, the boundary values are 199 (maximum of the lower invalid partition), 200, 500 and 501 (minimum of the upper invalid partition).

Neighbours depend on precision

The closest neighbour is one step away in the precision the specification uses. For whole numbers the step is 1; for amounts entered with two decimal places it is 0.01, so a valid range 1.50 to 25.00 has outside neighbours 1.49 and 25.01, not 1 and 26 counted in whole units.

2-value BVA tests each boundary value with its closest neighbour in the adjacent partition, while 3-value BVA tests each boundary value with both of its neighbours. The two CTFL variants differ only in the coverage items each boundary value produces:

In 3-value BVA a neighbour may sit in the same partition as the boundary value. The 2-value set is always contained in the 3-value set. Values two or more steps away from a boundary are not coverage items in either variant.

Why 3-value BVA is more rigorous

3-value BVA can detect defects that 2-value BVA misses. Suppose the requirement is x <= 10 but the code says x == 10. 2-value BVA tests 10 and 11: 10 behaves correctly under both conditions and 11 is correctly rejected, so both tests pass. 3-value BVA also tests 9, where the requirement says yes and the faulty code says no, so the defect is found. The extra neighbour inside the partition catches conditions that are only correct exactly at the boundary.

BVA coverage

BVA coverage is boundary values (coverage items) exercised divided by the total identified, for the variant being used. A test value inside an invalid partition that is not one of its boundary values or neighbours (say 3 when the range starts at 5) counts for EP but adds nothing to BVA coverage. Pass rates are results, not coverage.

Combining EP and BVA

A well-chosen BVA set also exercises every ordered partition, because each boundary value belongs to one. To reach both 100% EP and 100% 2-value BVA coverage, count the distinct boundary values (two per internal boundary) and then add one test value for each partition that has no boundaries, such as an unordered "not a number" partition. Open-ended outer partitions (below the minimum, above the maximum) still have one boundary value each, the one next to the valid range.

Decision table testing checks how the test object implements requirements in which different combinations of conditions result in different outcomes. It is an effective way to record complex logic such as business rules.

In a decision table, the conditions and the resulting actions of the system are rows, and each column is a decision rule: one unique combination of conditions together with the actions it triggers.

Notation

In a limited-entry table, conditions are true or false and actions are X or blank. In an extended-entry table, conditions and actions can take multiple values, such as ranges of numbers, equivalence partitions or discrete values. Dashes and infeasible combinations can occur in either kind.

To read a table for a given case, find the column whose condition values match; a dash matches any value. Then read its action cells: every X happens, every blank does not.

The coverage items are the columns containing feasible combinations of conditions, and 100% decision table coverage needs test cases that exercise all of them; infeasible combinations cannot be tested and are not coverage items. For a limited-entry table, the full table has one column per combination of condition values: with n Boolean conditions, 2 to the power n columns (2 conditions give 4, 5 give 32).

So the minimum number of test cases is the full column count minus the infeasible columns: with 32 columns of which 6 are infeasible, at least 26 test cases are needed. Covering each condition true and false once, or triggering each action once, is a weaker goal and can leave combinations untested.

Minimizing

The table can be simplified by deleting infeasible columns and merging columns in which some conditions do not affect the outcome into a single column, where those conditions get a dash. Work rule by rule: if one condition decides the outcome on its own (for example an expired membership blocks entry whatever else is true), all the columns sharing that value merge into one; columns where another condition still changes the action must stay separate. The minimized columns are then the coverage items.

Strengths and weaknesses

• Strength: a systematic approach to identify all combinations of conditions, some of which might otherwise be overlooked, and it helps to find gaps or contradictions in the requirements.
• Weakness: with many conditions, exercising all decision rules is time-consuming, because the number of rules grows exponentially with the number of conditions. To keep it manageable, use a minimized decision table or a risk-based approach to choose which rules to test.

A decision table answers "which combination leads to which action". Event sequences belong to state transition testing, and one value per group of equally processed values belongs to equivalence partitioning.

State transition testing models the behaviour of a system as a set of states and the transitions between them. A state diagram shows the possible states and the valid transitions. A transition is initiated by an event, may be qualified by a guard condition, is assumed to be instantaneous and may result in the software taking an action.

The common labelling syntax is event [guard condition] / action. In submit [formComplete] / sendConfirmation, submit is the event, formComplete is the guard that must hold for the transition to fire, and sendConfirmation is the action. Guard conditions and actions can be omitted when they do not exist or are irrelevant. The target state is shown by the arrow, not in the label.

State table

A state table is an equivalent model: rows represent states, columns represent events (together with guard conditions if they exist), and each filled cell represents a transition, holding the target state and the resulting action if one is defined. Unlike the diagram, the state table shows invalid transitions explicitly, as empty cells, which is why it is the natural source for all transitions coverage. A valid transition with no action, or a transition back into the same state, still names a target state in its cell.

A test case is usually a sequence of events that produces a sequence of state changes (and actions where needed). One test case can, and usually will, cover several transitions.

The CTFL syllabus defines three coverage criteria for state transition testing:

The subsumption runs one way only. Every valid transition ends in a state, so exercising all of them visits every state, but visiting every state does not exercise every transition when a state can be entered in several ways. Valid transitions coverage alone never attempts an invalid transition, so it cannot guarantee all transitions coverage.

Calculating coverage

For all transitions coverage, every cell of the state table is a coverage item: with 5 states and 2 events there are 10 cells. If 6 are valid and 4 invalid, exercising all 6 valid transitions and attempting 1 invalid one covers 7 of 10 items, 70%, while the same tests give 100% valid transitions coverage.

One invalid transition per test case

When testing invalid transitions, attempt only one invalid transition in each test case. This avoids defect masking (also called fault masking), where one defect hides another: if the first invalid event is handled wrongly, the system may end in an unexpected state, and later checks in the same test pass or fail for the wrong reason. Valid transitions are still used to reach the state from which the invalid event is attempted.

To find the minimum number of test cases for a criterion, trace event sequences from the initial state and use the structure of the model:

1. List the coverage items for the criterion: states, valid transitions, or all cells of the state table.
2. Count the endings. If every test starts in the initial state and stops when it reaches a final state, one test can only take one transition into a final state. The number of different transitions into final states (for valid transitions coverage), or of different final states (for all states coverage), is a lower bound.
3. Use loops. Transitions that lead back to an earlier state (such as retry or edit loops) let one test exercise many transitions before it ends, so the lower bound is often enough.
4. Write the sequences and check that together they cover every item.

For example, a model with three different transitions into final states needs at least three tests for valid transitions coverage, and three are enough if loops let the earlier tests pick up every other transition before they end. Without such loops, a transition that can only be reached on a path that bypasses another one forces an extra test. For all states coverage, the bound comes from the final states that cannot be reached together in one test.