CTFL flashcards

Fundamentals of Testing

• Testing = activities to discover defects and evaluate the quality of work products (test objects); it is more than running tests.
• Static testing examines work products without executing them and finds defects directly; dynamic testing executes the software and triggers failures.
• Verification checks against the specified requirements; validation checks against stakeholders' real needs in the operational environment — either can be static or dynamic.
• Typical test objectives include evaluating work products, finding defects, ensuring coverage, reducing risk, verifying requirements and compliance, informing decisions, building confidence and validating.
• Locating or correcting defects, proving there are no defects, and assigning fixes are not test objectives.
• Test objectives vary with the work product, test level, risks, SDLC and business context (corporate structure, competition, time to market).
• Debugging finds, analyses and removes the causes of failures: reproduce, diagnose, fix.
• A defect found by static testing is just removed — no reproduction or diagnosis.
• After a fix, confirmation testing (preferably by the original tester) checks the fix; regression testing checks nothing else broke.
• Testing is also planned, managed, estimated, monitored and controlled.

• Testing contributes in four ways: cost-effective defect detection, direct evaluation of quality for decisions, indirect representation of users, and meeting contractual, legal or regulatory requirements.
• Testing improves quality indirectly: the gain comes when debugging removes the defects testing found.
• Testers represent users because involving a representative set of real users throughout is usually too costly or impractical.
• Testing is a form of quality control. QC is product-oriented and corrective; QA is process-oriented and preventive and is everyone's responsibility.
• Other forms of QC include formal methods (model checking, proof of correctness), simulation and prototyping.
• Test results serve both: QC uses them to fix defects, QA uses them as feedback on development and test processes.
• Error (human action) leads to a defect (flaw in a work product), which may lead to a failure (an event at run time).
• Defects occur in documentation, code and supporting work products such as build files, and undetected early defects often spread into later work products.
• Some defects fail every time, some only in specific circumstances, some never; environmental conditions such as radiation can also cause defects in firmware.
• A root cause is the situation behind an error (time pressure, lack of training); addressing it prevents or reduces similar defects, while debugging fixes only the one defect.

• Testing shows the presence of defects; passing tests lower the probability of undiscovered defects but never prove there are none, whatever the coverage.
• Exhaustive testing is impossible except in trivial cases; focus effort with test techniques, test case prioritization and risk-based testing.
• Early testing saves time and money because defects removed early do not cause defects in derived work products and fewer failures occur later; start static and dynamic testing as early as possible.
• Defects cluster together in a few components; predicted and observed clusters are an important input to risk-based testing.
• Tests wear out when repeated unchanged; modify tests and data and write new ones — except where repetition is the point, as in automated regression testing.
• Testing is context dependent: there is no single test approach that suits every project.
• Absence-of-defects fallacy: verification against the specification is not enough — a system can still miss users' needs, business goals or competitors' standard; carry out validation too.
• Pesticide paradox is the older name for the tests-wear-out principle.

• The seven test activities are planning, monitoring and control, analysis, design, implementation, execution and completion; in practice they are iterative or parallel and tailored to the project.
• Monitoring compares actual progress with the plan; control takes action to meet the test objectives.
• Analysis decides what to test (prioritised test conditions, plus defect reports on the test basis); design decides how (test cases, charters, coverage items, data and environment requirements).
• Implementation creates what design specified: test data, procedures, suites, scripts, the execution schedule and a verified test environment with stubs and drivers.
• Execution compares actual with expected results, logs them and analyses anomalies; it produces test logs and defect reports.
• Completion turns open defects into change requests or backlog items, archives reusable testware, shuts down the environment, records lessons learned and issues the completion report.
• Context factors change the strategy, techniques, automation, coverage, documentation detail and reporting; project constraints are scope, time, budget and resources.
• Tracing test cases to requirements shows coverage; tracing test results to risks shows residual risk; traceability also supports change impact analysis, audits, governance and clearer reports.
• The test management role focuses on planning, monitoring and control, and completion; the testing role on analysis, design, implementation and execution. Roles are not job titles, and one person can hold both.

• Testing knowledge (e.g. test techniques) and analytical, critical and creative thinking increase effectiveness; technical knowledge (e.g. test tools) increases efficiency.
• Thoroughness, curiosity, attention to detail and being methodical help find defects, especially hard-to-find ones; domain knowledge helps communicate with end users and business representatives.
• Testers are bearers of bad news and confirmation bias makes people resist conflicting information, so defects are reported objectively and about the product, never the person.
• The whole team approach comes from Extreme Programming: anyone with the needed skills can do any task, everyone is responsible for quality, and the team shares a physical or virtual workspace.
• Its advantages are better team dynamics, enhanced communication and collaboration, and synergy from the team's combined skill sets.
• In a whole team, testers help business representatives create acceptance tests, agree test strategy and automation approach with developers, and transfer testing knowledge.
• Safety-critical contexts may need a high level of independence that the whole team approach alone does not give.
• Independence levels: author (none), same-team peers (some), outside the team but inside the organization (high), outside the organization (very high).
• Independence finds different kinds of defects and challenges assumptions, but risks isolation, developers losing responsibility for quality, and testers seen as a bottleneck.
• Independence does not replace familiarity; most projects test at multiple levels of independence.

Testing Throughout the Software Development Lifecycle

• The SDLC shapes the scope and timing of testing, documentation detail, techniques and approach, the extent of automation and the tester's role; it changes how testing is organised, not its principles.
• Sequential models: early static testing and test design, but dynamic testing waits for code. Iterative models: static and dynamic testing at all levels in every iteration, with fast feedback and extensive regression testing.
• Agile favours lightweight documentation, extensive automation for regression, and experience-based manual testing.
• Every SDLC: a test activity for each development activity, distinct objectives per test level, test analysis and design during the corresponding development phase, and reviews of drafts as soon as they exist.
• TDD (test, code, refactor), ATDD (tests from acceptance criteria before development) and BDD (Given/When/Then, then automated) all define tests before the code.
• DevOps brings fast feedback, CI with component tests and static analysis, stable environments, more non-functional focus and lower regression risk, but the pipeline, tools and automation must be built and maintained.
• Manual testing from the user's perspective is still needed in DevOps.
• Shift left tests earlier without neglecting later testing; it may cost more early, is expected to save later, and needs stakeholder buy-in.
• Retrospectives involve the whole team, ask what to keep, what to improve and how, are recorded normally in the test completion report, and must be followed up.

• There are five test levels: component, component integration, system, system integration and acceptance; each is an instance of the same test process.
• Levels are distinguished by test object, objectives, test basis, defects and failures, and approach and responsibilities, not by quality characteristic.
• In sequential models one level's exit criteria often feed the next level's entry criteria; in iterative models levels can overlap.
• Component testing isolates a component with harnesses, frameworks and stubs and is normally done by developers in their own environment.
• Component integration testing looks at interfaces inside one system and depends on the integration strategy; system integration testing looks at interfaces with other systems in an operational-like environment.
• Acceptance testing validates business needs and readiness for deployment, ideally by intended users; its forms are UAT, OAT, contractual, regulatory, alpha (developer's site) and beta (users' own sites).
• Functional testing checks what the system does (completeness, correctness, appropriateness); non-functional testing checks how well, should start early and may need special environments such as a usability lab.
• Black-box tests come from external specifications; white-box tests come from internal structure (code, architecture, workflows, data flows) and aim at coverage.
• All four test types can be applied at all test levels, with a different focus at each.
• Confirmation testing checks the original defect is fixed; regression testing checks a change caused no adverse effects anywhere, including connected systems and the environment, after an impact analysis.

• Maintenance testing tests changes to an operational system — to its code, its environment, or its retirement.
• Maintenance categories follow the reason for the change: corrective (fix a defect), adaptive (fit a changed environment), improving performance, improving maintainability (ISO/IEC 14764).
• Maintenance ships as planned, release-based deployments or as unplanned deployments (hot fixes); delivery speed is not one of the scope factors.
• Impact analysis may be done before a change, to help decide whether to make it based on its consequences in other areas.
• Testing a production change evaluates the success of the change and checks for regressions in unchanged parts — usually most of the system.
• Scope depends on the degree of risk of the change, the size of the existing system and the size of the change.
• Triggers: modification (enhancements, corrective changes, hot fixes); upgrade or migration of the operational environment (tests of the new environment and changed software; data conversion when another application's data is migrated in); retirement (data archiving; restore and retrieval).
• An environment-only change to a live system, such as a hardware or platform change with no new features, is still a maintenance testing trigger.

Static Testing

• Static testing examines work products without executing the software; its two forms are manual reviews and tool-supported static analysis.
• Its objectives are improving quality, detecting defects and assessing readability, completeness, correctness, testability and consistency; it serves both verification and validation.
• Almost any readable work product can be reviewed, including contracts, test charters and backlog items; third-party executable code is the exception, for legal reasons.
• Static analysis needs a formal structure to check against (models, code, text with a formal syntax); plain-text work products are reviewed instead.
• Static testing finds defects in the earliest lifecycle phases, which fulfils the early testing principle and lowers the cost of fixing them.
• Only static testing finds defects that cause no failure: unreachable code, design patterns not implemented as desired, defects in non-executable work products.
• Reviews cost effort, but total project costs are usually much lower than without them; static analysis in CI usually means fewer code defects and lower development effort.
• Static testing finds defects directly and applies to non-executable work products; dynamic testing causes failures, needs executable work products and measures execution-dependent characteristics such as performance efficiency.

• Early and frequent stakeholder feedback surfaces quality problems early, prevents requirement misunderstandings, gets changes in sooner and keeps the team on the most valuable, risk-reducing features.
• Too little stakeholder involvement risks a product that misses the stakeholders' original or current vision, with costly rework, missed deadlines, blame games or project failure.
• The review process runs planning, review initiation, individual review, communication and analysis, then fixing and reporting.
• Planning sets the scope and exit criteria; initiation makes sure participants have access, know their roles and have what they need.
• Anomalies are not necessarily defects: their status, owner and actions are decided in communication and analysis; defect reports are raised in fixing and reporting.
• Manager: what is reviewed and resources. Review leader: overall responsibility, who, when, where. Moderator: runs the meeting. Scribe: collates and records. Author: creates and fixes.
• Walkthroughs are led by the author with many objectives; technical reviews are led by a moderator to reach technical decisions; inspections are the most formal, aim at maximum anomalies, collect metrics, and the author cannot be review leader or scribe.
• Formality depends on SDLC, process maturity, criticality and complexity, regulation and audit-trail needs; the same work product can get several review types.
• Successful reviews have clear objectives and measurable exit criteria, never evaluate participants, use small chunks, preparation time, feedback, management support, training and facilitated meetings.

Test Analysis and Design

• Test techniques support test analysis (what to test) and test design (how to test).
• Because exhaustive testing is impossible, techniques derive a relatively small but sufficient set of test cases systematically.
• Techniques help define test conditions, identify coverage items and identify test data.
• The syllabus has three categories: black-box (specification-based), white-box (structure-based) and experience-based.
• Black-box tests analyse specified behavior, can be designed before code exists and stay useful when only the implementation changes.
• White-box tests analyse internal structure, can only be created after the design or implementation exists and change with it.
• Experience-based effectiveness depends heavily on tester skill; these techniques complement, not replace, the systematic ones.
• EP, BVA, decision tables and state transitions are black-box; statement and branch testing are white-box; error guessing, exploratory and checklist-based testing are experience-based.
• Collaboration-based test approaches are a separate approach, not a fourth technique category.

• Equivalence partitions must not overlap and must be non-empty; they can be identified for any data element (inputs, outputs, configuration, internal and time-related values) and can be ordered or unordered.
• A valid partition holds values the test object should process and an invalid partition values it should ignore or reject (definitions can vary between teams); 100% EP coverage needs every partition, valid and invalid, exercised at least once.
• Each Choice coverage needs each partition of each set exercised once, without combinations, so the minimum test count is the size of the largest set.
• BVA applies only to ordered partitions; a partition's minimum and maximum are its boundary values, and neighbours are one step away in the specified precision.
• 2-value BVA tests each boundary value and its closest neighbour in the adjacent partition; 3-value BVA tests both neighbours and can catch defects 2-value misses.
• In a decision table each column is a rule; a dash means irrelevant, N/A means infeasible, X means the action occurs and a blank means it does not.
• Decision table coverage items are the feasible columns; the table grows exponentially, so minimize it (drop infeasible, merge irrelevant) or select rules by risk.
• State transition labels read event [guard] / action; a state table shows invalid transitions as empty cells.
• Valid transitions (0-switch) coverage subsumes all states coverage; all transitions coverage subsumes both and is the minimum for mission- and safety-critical software.
• Attempt only one invalid transition per test case to avoid defect masking.

• Statement coverage = executable statements exercised ÷ total executable statements; branch coverage = branches exercised ÷ total branches; both count each item once across the whole suite.
• 100% statement coverage means every executable statement ran at least once, so a defective statement was executed and may have caused a failure — not that it did.
• Data-dependent defects (a calculation that is wrong only for certain input values) can survive 100% statement and branch coverage.
• A branch is a transfer of control between two nodes; unconditional (straight-line) and conditional (decision outcome) branches are both coverage items.
• 100% branch coverage takes every decision outcome: true and false for IFs, every case including the default for CASE, continue and exit for loops.
• Branch coverage subsumes statement coverage; 100% branch with less than 100% statement coverage is impossible.
• Full branch coverage can still miss defects that need a specific combination of branches (a path).
• White-box testing considers the whole implementation, so it works with vague or outdated specifications, but it may miss requirements that were never implemented.
• White-box techniques can be used statically (dry runs of code or pseudocode) and give an objective code coverage measure that black-box testing alone does not.

• Error guessing anticipates errors, defects and failures from how the application worked before, the errors developers tend to make and failures in similar applications.
• Anticipated errors fall into input, output, logic, computation, interfaces and data.
• A fault attack is methodical error guessing: build a list of possible defects and failures, then design tests to identify, expose or cause them.
• Exploratory testing designs, executes and evaluates tests at the same time while learning about the test object.
• Session-based testing adds a time-box, a charter whose objectives act as high-level test conditions, a session sheet and a debriefing — but sessions are still not repeatable like scripts.
• Exploratory testing suits sketchy specifications and time pressure, complements formal techniques, can incorporate them, and works best with experienced, curious, analytical testers.
• Checklist items are questions, each checkable separately and directly; leave out automatable items, entry/exit criteria and overly general items.
• Checklists lose effectiveness as developers learn; add items for new high-severity defects and update from defect analysis without letting the list grow too long.
• Specific items give consistency and repeatability; high-level items give variability and potentially greater coverage.

• A user story has 3 Cs: Card (the medium), Conversation (how the software will be used, documented or verbal) and Confirmation (the acceptance criteria).
• Story template: As a [role], I want [goal], so that [resulting business value for the role].
• Writing stories with business, development and testing gives a shared vision of what to deliver; brainstorming and mind mapping help.
• INVEST: Independent, Negotiable, Valuable, Estimable, Small, Testable.
• Stakeholders who cannot say how to test a story signal an unclear story, a story not valuable to them, or a need for help with testing.
• Acceptance criteria are the conditions for stakeholders to accept the implementation; they set scope, build consensus, cover positive and negative scenarios, are the basis for acceptance testing and enable estimation.
• Given/When/Then is scenario-oriented; a verification list or an input-output table is rule-oriented; any other format works if well-defined and unambiguous.
• ATDD starts with a specification workshop; tests are written before implementation by the team or a tester, in natural language with preconditions, inputs and postconditions.
• ATDD order: typically positive tests, then negative tests, then non-functional characteristics.
• ATDD tests cover every characteristic of the story, nothing beyond it, and no two tests cover the same characteristic; automated, they become executable requirements.

Managing the Test Activities

• A test plan documents the means and schedule for achieving the test objectives and communicates them; reports describe testing already done.
• The test approach holds test levels and types, techniques, deliverables, entry/exit criteria, independence, metrics, test data and environment needs, and deviations from policy or strategy.
• Release planning: testable stories and acceptance criteria, project and quality risk analyses, test approach and release test plan. Iteration planning: detailed story risk analysis, testability, testing tasks and their estimates.
• Entry criteria are preconditions to start; exit criteria are what must be achieved to finish. Skipping entry criteria makes testing more difficult, time-consuming, costly and risky.
• Running out of time or budget is a valid exit only when stakeholders have reviewed and accepted the risk.
• Agile: Definition of Ready = entry criteria for a user story; Definition of Done = exit criteria for a releasable item.
• Ratios use the organization's own past projects; extrapolation uses the current project's early data; Wideband Delphi (and Planning Poker) iterates expert estimates to consensus.
• Three-point estimation: E = (a + 4m + b) / 6, SD = (b − a) / 6, result E ± SD.
• Additional coverage prioritization picks each next test by the new coverage it adds; dependencies and resource windows can override priority order.
• Test pyramid: many small, fast, isolated tests at the bottom, few slow end-to-end tests at the top. Quadrants: Q1 component, Q2 functional/story, Q3 exploratory/usability/UAT, Q4 smoke and non-functional except usability.

• Risk management = risk analysis (identification + assessment) and risk control (mitigation + monitoring); risk-based testing selects, prioritizes and manages test activities on that basis.
• A risk has two attributes: likelihood (probability greater than zero and less than one) and impact (the harm). Together they give the risk level.
• Quantitative risk level = likelihood × impact; qualitative uses a risk matrix; the two can be mixed. The higher the level, the more important the treatment.
• Project risks concern managing and controlling the project and hit schedule, budget or scope; product risks concern quality characteristics and hit users, revenue, reputation, third parties and safety.
• Project risk categories: organizational (delays, inaccurate estimates, cost-cutting), people (skills, conflicts, communication, staff shortage), technical (scope creep, poor tool support), supplier (delivery failure, bankruptcy).
• Product risk analysis aims to minimize residual product risk by focusing test effort, and should start early in the SDLC.
• Its results drive test scope, levels, types, techniques, coverage, effort estimates, prioritization, and whether non-testing activities could also reduce risk.
• Responses to product risks: mitigation by testing, acceptance, transfer and contingency plan; they can be combined.
• Mitigation by testing: suitably skilled testers, appropriate independence, reviews and static analysis, suitable techniques and coverage, test types for the affected quality characteristic, dynamic and regression testing.
• Risk monitoring checks mitigation is effective, improves the assessment and identifies emerging risks.

• Test monitoring gathers information to assess progress and whether exit criteria are met; test control acts on it with control directives; test completion consolidates experience and testware at milestones.
• Control directives match triggers: reprioritize tests when a risk becomes an issue, re-evaluate entry/exit criteria after rework, adjust the schedule for a late test environment, add resources where testing falls behind.
• Test completion happens when a test level is completed, an agile iteration finishes, a test project is completed or cancelled, a system is released or a maintenance release is completed.
• Test metrics show progress against schedule and budget, the current quality of the test object and the effectiveness of testing.
• Seven metric categories: project progress (incl. test effort), test progress (incl. test execution time), product quality (e.g. mean time to failure), defect (e.g. defect detection percentage, defect density), risk (residual risk), coverage (requirements, code) and cost (cost of testing, cost of quality).
• Progress reports support ongoing control and are produced regularly; they include the period, progress and deviations, impediments, metrics, new and changed risks and the next period's testing.
• Completion reports summarize a finished stage using the progress reports; only they contain the evaluation against the plan's exit criteria, unmitigated risks and unfixed defects, and lessons learned.
• Audiences shape report content, formality and frequency; ISO/IEC/IEEE 29119-3 gives templates for test status (progress) and test completion reports.
• Status can be communicated verbally, by dashboards, by email or chat, by online documentation or by formal test reports; the choice depends on test management concerns, the test strategy, regulatory standards or the team, more formal communication may suit distributed teams, and each stakeholder gets the information they need.

• Configuration management identifies, controls and tracks work products as configuration items; it is about versions of work products, not risks, defects or progress.
• Testware (test plans, test strategies, test conditions, test cases, test scripts, test results, test logs, test reports) and test items are all configuration items.
• A configuration item is an aggregation of work products designated for CM and treated as a single entity.
• For a complex item such as a test environment, CM records its constituent items, their relationships and their versions.
• A configuration item approved for testing becomes a baseline, a formally approved version that can only be changed through formal change control.
• CM records the changed items when a new baseline is created, so the team can revert to a previous baseline and reproduce previous test results.
• CM ensures all configuration items, including test items, are uniquely identified, version controlled, tracked for changes and related to each other for traceability.
• CM ensures documentation and software items are referenced unambiguously in testware: every reference names the item's identifier and version.
• In CI/CD DevOps pipelines, automated configuration management is normally included in the pipeline; frequent builds make CM more important, not less.

• The defect management workflow is: log the anomaly, analyze and classify it, decide on a response (fix or keep as it is), then close the report.
• At a minimum, the process includes a workflow from discovery to closure and rules for classification, and all involved stakeholders follow it.
• A reported anomaly may be a real defect, a false positive or a change request; this is resolved while the report is handled.
• Defects found by static testing, especially static analysis, are advisably handled in a similar way to those found by dynamic testing.
• A defect report aims to give enough information to resolve the issue, to track work-product quality and to provide ideas for improving the development and test process.
• A typical report has an identifier, title, date, organization and author with role, test object and environment, context, a reproducible failure description, expected and actual results, severity, priority, status and references.
• A defect report records the anomaly as observed; information found while resolving it, such as debugging results, is added later by those who resolve it.
• Severity is the degree of impact on stakeholders' interests or requirements; priority is the urgency to fix; they vary independently.
• Defect tools can fill in the identifier, date, author and initial status automatically.
• ISO/IEC/IEEE 29119-3 provides templates for defect reports and calls them incident reports.

Test Tools

• Test tools support and facilitate test activities; the syllabus categories are examples, not a closed list.
• Management tools increase test process efficiency by managing the SDLC, requirements, tests, defects and configuration, including traceability and version control of testware.
• Static testing tools support reviews and static analysis of any work product without executing it.
• Test design and implementation tools generate test cases, test data and test procedures; producing testware, not running it, is their job.
• Test execution and coverage tools run tests automatically and measure coverage during execution.
• Non-functional testing tools make possible testing that is difficult or impossible manually, such as large-scale load generation or security scanning.
• DevOps tools support the delivery pipeline, automated builds and CI/CD; collaboration tools support communication.
• Virtual machines and containerization tools support scalability and deployment standardization of test environments.
• Any tool that assists in testing is a test tool in that context, for example a spreadsheet used to track test results.

• Simply acquiring a tool does not guarantee success; lasting benefits need effort for introduction, maintenance and training, and risks need analysis and mitigation.
• Time saved: less repetitive manual work — running regression tests, re-entering test data, comparing expected and actual results, checking coding standards.
• Consistency: fewer simple human errors because tests are derived consistently, test data is created systematically and tests run in the same order and frequency.
• Objective assessment: precise measures such as coverage, including measures too complicated for humans to derive.
• Reporting support: statistics, graphs and aggregated data on test progress, failure rates and execution duration for test management.
• Reduced execution time gives earlier defect detection, faster feedback and faster time to market; freed testers can design new, deeper tests.
• Risks: unrealistic expectations, inaccurate estimation of effort, automating where manual testing fits better, over-reliance on the tool, vendor dependency, abandoned or fast-changing open source, platform incompatibility, regulatory non-compliance.
• Commercial and open-source tools carry different risks: vendor dependency for one, abandonment or quite frequent component updates for the other.

Yes — the whole CTFL deck is free and needs no account, as is every CTFL revision lesson. Create a free account to save progress and drill practice questions; full-length mock exams are a one-time unlock per certification.