ISTQB CTFL Collaboration-based Test Approaches: User Stories, Acceptance Criteria and ATDD

Collaboration-based test approaches are ways of designing tests in which business representatives, developers and testers work together to avoid defects rather than only to find them. The other test techniques in this part of the syllabus (black-box, white-box and experience-based) focus on detecting defects; collaboration-based approaches focus on defect avoidance through collaboration and communication, by making sure everyone understands the same thing before anything is built.

The CTFL syllabus covers three linked practices:

• Collaborative user story writing (FL-4.5.1): the business, development and testing perspectives write the story together.
• Acceptance criteria (FL-4.5.2): the conditions that decide whether the story's implementation is accepted, written in a scenario-oriented, rule-oriented or other clear format.
• Acceptance test-driven development (ATDD) (FL-4.5.3): a test-first approach that derives test cases from the story and its acceptance criteria before the code is written.

Where ATDD sits among the other test-first approaches (TDD and BDD) is covered in the lesson on testing in the SDLC; this lesson is about doing it.

A user story represents a feature that will be valuable to a user or purchaser of a system or software. A user story has three critical aspects, known as the 3 Cs:

The acceptance criteria are the content of the confirmation, not a separate fourth item.

The most common format for the story itself is:

As a [role], I want [goal to be accomplished], so that I can [resulting business value for the role].

A complete story statement has all three parts: who wants it, what they want, and why it is valuable to them. For example, "As a nurse, I want to see today's medication schedule" names a role and a goal but not why it matters, so it is incomplete; adding "so that I can give each dose on time" supplies the value. The final "so that" part is always the business value for the role. The template is followed by the acceptance criteria.

Writing user stories collaboratively gives the team a shared vision of what should be delivered, because each participant contributes their own perspective:

• Business representatives bring what is valuable and why.
• Development brings what is feasible and how much effort it needs.
• Testing brings how the result will be checked, including the cases nobody mentioned.

Techniques such as brainstorming and mind mapping can be used for this collaborative authorship.

A story written from only one perspective tends to miss what the others would have spotted: whether the value is clear, whether it is feasible, and how anyone will check it. Writing the story together and agreeing its acceptance criteria up front gives everyone the same picture of "done", which makes the story testable and gives the team a common basis for estimation.

Good user stories meet the INVEST criteria:

The T stands for Testable. A story that asks for monthly reports to be "easy to read" is a typical failure: with no agreed measure, nobody can say how to confirm it is done.

When stakeholders cannot say how to test a story

If stakeholders do not know how to test a user story, the syllabus gives three possible reasons:

• the story is not clear enough;
• it does not reflect something valuable to them; or
• the stakeholders simply need help with testing.

Read the situation to choose between them: if the stakeholders already agree the story is clear and valuable, the remaining explanation is that they need help with testing.

Acceptance criteria for a user story are the conditions that an implementation of the user story must meet to be accepted by stakeholders. From this perspective they can be viewed as the test conditions that should be exercised by the tests. They are usually a result of the conversation.

They belong to one user story. The acceptance criteria may also detail some of the issues described in the story. The syllabus lists what acceptance criteria are used to do:

• define the scope of the user story (its boundaries);
• reach consensus among the stakeholders;
• describe both positive and negative scenarios;
• serve as a basis for user story acceptance testing;
• allow accurate planning and estimation.

Acceptance criteria say what the implementation must achieve to be accepted; they do not prescribe how developers implement the logic or design the code. They support the conversation rather than replace it, and they are not a place to record defects.

There are several ways to write acceptance criteria for a user story. The syllabus names the two most common formats and allows others:

Both a bulleted list and an input-output table are rule-oriented: the table is still a set of rules, just tabulated. Given/When/Then is always scenario-oriented. A custom format is acceptable as long as the acceptance criteria are well-defined and unambiguous. Clarity is the test, not the layout.

Acceptance test-driven development (ATDD) is a test-first approach: test cases are created prior to implementing the user story. They are created by team members with different perspectives, such as customers, developers and testers. ATDD runs in this order:

1. Specification workshop. The first step: the user story and (if not yet defined) its acceptance criteria are analysed, discussed and written by the team. Incompleteness, ambiguities and defects in the story are resolved during this process, and the resolution goes into the acceptance criteria. If, for example, the team notices a shared shopping-list story never says what happens when two family members edit the list at once, they agree the behaviour in the workshop and write it into the criteria.
2. Create the test cases. This can be done by the team as a whole or by a tester individually. Either way it happens before coding. The test cases are based on the acceptance criteria and can be seen as examples of how the software works, which helps the team implement the story correctly.

Implementation of the feature then follows; if the tests are automated, developers write the supporting automation code during that implementation.

Since examples and tests are the same, the terms are often used interchangeably. During test design the test techniques described in the rest of this domain (equivalence partitioning, boundary value analysis and so on) may be applied.

How ATDD tests are expressed

Typically the test cases are written in natural language that stakeholders can understand, containing the necessary preconditions (if any), the inputs and the postconditions. They are not unit-level code tests, which is what TDD produces. ATDD tests may be executed manually or automated. When the tests are captured in a format supported by a test automation framework, developers can automate them by writing supporting code; the acceptance tests then become executable requirements.

ATDD test cases are typically written positive first, then negative, then non-functional:

1. Positive tests first: they confirm the correct behaviour without exceptions or error conditions, covering the sequence of activities executed when everything goes as expected.
2. Negative tests next: invalid inputs, rejected requests and error conditions.
3. Non-functional characteristics last: for example performance efficiency (such as an agreed response time) or usability.

For a document upload story, that means "a PDF within the size limit is uploaded and listed" (positive), then "a file over the size limit is rejected with a message" (negative), then "an upload completes within the agreed time" (non-functional).

Coverage rules

The test cases must:

• cover all the characteristics of the user story;
• not go beyond the story: a test for behaviour the story and its criteria do not mention (an export, an email, a future extension) does not belong;
• not duplicate: no two test cases should describe the same characteristic of the story.

So when reviewing a draft set, drop any test that repeats a characteristic already covered (for a gym class booking story, booking a different class that has free places is still the same characteristic as booking one), drop any test outside the story, and add a test for any characteristic nobody has covered yet (cancelling a booking, if the criteria allow it), rather than another variation of one already tested.

Deriving ATDD test cases (the K3 objective) means turning each acceptance criterion into at least one concrete example with a precondition, input and expected result, then checking the set against the order and coverage rules.

Each criterion gets its expected result from the criteria

Criteria for a single-use coupon: Given the coupon has not been used, When the customer applies it, Then 10% is taken off the order and the coupon is marked used. Given the coupon has already been used, When the customer applies it, Then it is rejected and the order total is unchanged.

A negative test is defined by an input or state that violates the condition and by an expected result taken from the criteria. The tester does not invent the outcome; the criteria state it.

Applying boundary value analysis inside ATDD

Criteria: customers aged 18 or older may buy the product; younger customers are refused. Ages are whole years. "Or older" puts 18 in the allowed partition, so the boundary sits between 17 (largest refused value) and 18 (smallest allowed value). With 2-value boundary value analysis the tests are 17 (refused) and 18 (allowed): one test per criterion, each on its own side of the boundary. Reading an inclusive threshold ("or more", "or older", "at least") correctly is what places the boundary value in the right partition.

Each test should fail if its rule were missing

Criteria: orders of 100 or more get a 10% discount; the discount is recalculated whenever an item is removed. "Basket of 120: discount applied" covers the first rule. For the recalculation rule, a useful test is "basket of 110 with a discount, remove an item worth 20: total 90 with no discount": without recalculation the discount would stay, so the test reveals whether the rule is implemented. A test such as "basket of 60, remove an item: no discount" would pass whether or not recalculation exists, so it proves nothing about that rule.