ISTQB CTFL Test Levels, Test Types, and Confirmation vs Regression Testing

A test level is a group of test activities that are organised and managed together, and each level is an instance of the test process carried out on the software at a particular stage of development. The same activities (planning, analysis, design, implementation, execution, completion) happen at every level; what changes is what is being tested and why.

The CTFL syllabus names five test levels:

1. Component testing (also called unit testing)
2. Component integration testing (also called unit integration testing)
3. System testing
4. System integration testing
5. Acceptance testing

To keep levels from overlapping in purpose, they are distinguished by a set of attributes (the list is not exhaustive):

• Test object – what is tested (a component, a group of components, the whole system, its interfaces with other systems).
• Test objectives – what the level is trying to achieve.
• Test basis – the work products the tests are derived from.
• Defects and failures – the kinds of problems the level typically finds.
• Approach and responsibilities – how the level is carried out and who does it.

Notice what is not on that list: quality characteristics such as performance or usability. Those distinguish test types, which cut across every level (see below).

How levels relate to the lifecycle

In sequential development models, the exit criteria of one level often form part of the entry criteria for the next, so the levels run as a chain. In some iterative models this does not apply: development activities can span several test levels, and test levels can overlap in time. How the lifecycle as a whole shapes testing is covered in the lesson on testing in the context of an SDLC.

Component testing checks components in isolation, and component integration testing checks the interfaces and interactions between those components. Both look inward at the parts of a single system.

Component testing

The test object is one component (a class, a module, a function) separated from the rest of the system. Isolation usually needs specific support: test harnesses and unit test frameworks, with dependencies such as a database or another service replaced by stubs or other test doubles. Component testing is normally performed by developers in their own development environments. Test doubles let a component be exercised without the components it depends on; checking how the real components then work together is the job of the next level.

Component integration testing

The test object is the interface between components: does one module pass the right data, in the right format, at the right time, to another? This level depends heavily on the integration strategy, which sets the order in which components are combined:

• Bottom-up – start with lower-level components and work upwards, using drivers to call them.
• Top-down – start with higher-level components and work downwards, using stubs for components not yet integrated.
• Big-bang – integrate everything at once, which makes interface failures hard to localise.

Component integration testing comes before system testing in a sequential model, because the system's own components must work together before the system as a whole can be evaluated.

System testing evaluates the overall behaviour and capabilities of the whole system, while system integration testing evaluates the interfaces between that system and other systems or external services.

System testing

The test object is the entire system or product. System testing often includes functional testing of end-to-end tasks and non-functional testing of quality characteristics; some non-functional characteristics, such as usability, are best tested on the complete system in a representative test environment. Simulations of sub-systems can also be used. The test basis is the system's specifications, such as system requirements and use cases, and system testing may be performed by an independent test team.

System integration testing

The test object is the connection between the system under test and the outside world: an external SMS gateway, a partner's API, a downstream reporting system. Its test basis includes the agreed interface specifications. Because these interfaces behave differently in toy setups, system integration testing needs suitable test environments, preferably similar to the operational environment.

Acceptance testing focuses on validation and on demonstrating readiness for deployment, which means showing that the system fulfils the users' business needs. Ideally it is performed by the intended users.

The difference from system testing is the question asked. System testing asks whether the system meets its specifications (verification), and it covers both functional and non-functional behaviour. Acceptance testing asks whether this is the right system for the people who will use and run it, and whether it can go live. It is not "the non-functional level": both levels can include non-functional testing.

The syllabus lists the main forms of acceptance testing (it groups contractual and regulatory acceptance testing together):

Alpha and beta are the closest twins: both gather early feedback from users on pre-release software, and the deciding difference is where the testing happens.

Each level differs in test object, typical test basis and who usually does the work, which is the quickest way to classify a scenario.

To classify a scenario, start with the test object (one isolated component, an interface inside the system, the whole system, an interface to another system, or the system's fitness for its users and operators), then check that the test basis and the people involved fit the same level.

A test type is a group of test activities aimed at a particular quality characteristic or test basis, and the CTFL syllabus names four: functional, non-functional, black-box and white-box testing. Functional and non-functional testing differ in what is evaluated.

Functional testing: what the system does

Functional testing evaluates the functions that a component or system should perform. Its main objective is to check three functional sub-characteristics:

• Functional completeness – are all the specified functions present?
• Functional correctness – do the functions give correct results?
• Functional appropriateness – do the functions help users accomplish their specified tasks and objectives?

Non-functional testing: how well the system does it

Non-functional testing evaluates attributes other than functional characteristics. Its objective is to check the non-functional quality characteristics of the ISO/IEC 25010 product quality model, such as performance efficiency, compatibility, usability, reliability, security, maintainability and portability. (The 2023 revision of ISO/IEC 25010 renames usability as interaction capability and portability as flexibility, and adds safety; the ideas are the same.)

Four points about non-functional testing are worth knowing well:

• Start early. It is sometimes appropriate for non-functional testing to start early in the lifecycle, for example as part of reviews and component testing. Late discovery of non-functional defects can pose a serious threat to a project's success, so leaving it all to the last week before release is a risk, not a plan.
• Derived from functional tests. Many non-functional tests reuse functional tests and add a constraint: perform the function and check it completes within a specified time.
• Special environments. Some non-functional testing needs a very specific environment, such as a usability lab for usability testing.
• Not tied to one level. It is not reserved for acceptance testing, end users or production.

Black-box and white-box testing differ in where the tests come from: black-box tests are derived from documentation external to the test object, and white-box tests are derived from its internal structure.

"Structure" is wider than code. A tester at system level who designs tests to cover every interface in the architecture diagram, every step of a business workflow or every data flow between subsystems is doing white-box testing, because the tests come from structure and aim at coverage. Equally, black-box testing is not limited to the higher levels: a developer can derive component tests from the component's specification.

Tests based on the tester's knowledge of past defects and likely failure areas are a third category, experience-based testing, covered with test techniques in a later domain.

Yes. All four test types can be applied at all test levels, although the focus differs at each level, and different test techniques can be used to derive test conditions and test cases for each type.

Treat levels and types as two independent dimensions: the level says at what stage and on what object you test, the type says which characteristic or basis you test against. Every combination is possible:

When asked to classify a test that combines both, identify the level from the test object, then the type from the basis or characteristic, and check that both hold.

Confirmation testing checks that an original defect has been successfully fixed; regression testing checks that a change, including that fix, has caused no adverse consequences. Both are change-related testing, and both are needed at every test level where defects are fixed or changes are made.

Ways to do confirmation testing

Depending on risk, a fixed version can be tested by executing all test cases that previously failed because of the defect, and also by adding new tests to cover any changes made to fix it. When time or money for fixing defects is short, confirmation testing can be restricted to simply exercising the steps that should reproduce the failure and checking that it no longer occurs.

Scope of regression testing

Adverse consequences of a change can affect the component that was changed, other components in the same system, or even other connected systems. Regression testing is not restricted to the test object either: a change to the environment (a new application server, operating system or database version) calls for regression testing even when the application code is unchanged. Confirmation testing does not fit that case, because there is no fixed defect to confirm.

Impact analysis first

Before regression testing, it is advisable to perform an impact analysis to optimise its extent. Impact analysis shows which parts of the software, and which connected systems, could be affected by the change, so the team runs the regression tests that matter: wider than the changed module alone, narrower than the whole suite by default.

Automation and continuous integration

Regression test suites are run many times, and the number of regression test cases generally grows with each iteration or release, so regression testing is a strong candidate for automation, which should start early in the project. Where continuous integration is used, as in DevOps, it is good practice to include automated regression tests, possibly at several test levels.

After a defect fix, the two types are used together: confirmation testing first checks the fix, then regression testing checks for side effects of that fix. A fix that has passed confirmation testing still needs regression testing, because fixing one defect can introduce another.

A typical sequence after a defect in a shared component is fixed:

1. Confirm the fix – re-run the failed test (at the level where the failure was found), add tests for the changed code where risk justifies it, or, if time is short, at least repeat the reproduction steps.
2. Analyse the impact – identify the other components and connected systems that use the changed component.
3. Run regression tests on the areas the analysis selects, including connected systems, not just the changed component.

Keep the two jobs apart. A regression suite is chosen to detect side effects, so it may never exercise the original defect; showing that the defect is gone is the job of confirmation testing. The reach of regression testing follows the impact of the change, which can extend to every system that consumes the changed component's output.

Changes that arise after a system is in operation, and how maintenance testing scopes them, are covered in the maintenance testing lesson.