CT-GenAI: Applying Prompt Engineering to Test Analysis, Design, Regression and Monitoring

Generative AI supports software test tasks when a tester applies prompt engineering to a specific test activity: test analysis, test design and implementation, automated regression testing, and test monitoring and control. CT-GenAI section 2.2 describes what an LLM can do in each of these activities, which inputs it needs, and which prompting technique (prompt chaining, few-shot prompting or meta prompting) fits the task. The techniques themselves are defined in section 2.1; this lesson is about applying them.

One principle runs through the whole section: high-quality input is crucial for meaningful GenAI results. The quality and relevance of what you give the LLM, in relation to the task, directly decides the accuracy and precision of what comes back. Most practical improvements in this topic come from one of three moves:

• Give the data the task depends on. Risk information for prioritization, any visual or structured material the details live in, the documentation of the assets a script must reuse, and the constraints a plan must respect.
• Ask for the result you actually need. A defined output format, an artefact the tester can check, and an explicit statement of what the result is for.
• Check the result in proportion to risk. The syllabus warns that GenAI can make mistakes, so generated output must be checked carefully depending on the associated risk. A plausible-looking answer is not a verified one.

In test analysis, GenAI takes the test basis (requirements, user stories, technical specifications, GUI wireframes and other relevant information) and produces typical test analysis work products, above all prioritized test conditions such as acceptance criteria. Test execution schedules, scripts and completion reports belong to later activities.

Identifying potential defects in the test basis

An LLM can analyse requirements and user stories for inconsistencies, ambiguities and incomplete information that could lead to defects: statements that contradict each other, behaviour left undefined, terms that cannot be measured. It does this by comparing similar requirement patterns and by applying knowledge from previous defect reports; giving the LLM defect reports from earlier releases of the same product helps it recognise the kinds of requirement that caused trouble before. A useful finding both flags the problem and suggests an improvement: it names what is vague, missing or contradictory and proposes wording that a tester could verify. For example, a statement that an export "supports the usual formats" leaves the formats undefined, so a good finding asks for the list of formats to be written down. This review happens before tests are designed, which is why it prevents defects rather than detecting them.

Generating test conditions

Using natural language processing, an LLM can interpret requirements and user stories and break them down into measurable, testable statements, which become test conditions or acceptance criteria. The output can only be as complete as the input. When some of the details the criteria depend on exist outside the text, such as a screen mock-up, a diagram or a table in a specification, that material belongs in the prompt as input data; an image can be supplied directly, which makes it a multimodal prompt. Whatever the LLM has not been shown, it can only guess. Hands-on objective 2.2.1a practises exactly this: a structured prompt (role, context, instruction, textual and image input data, constraints, output format) to produce well-formed, clear and complete acceptance criteria, with extra context such as field constraints or business rules where relevant.

Hands-on objective 2.2.1b adds prompt chaining with human verification to refine acceptance criteria step by step: first identify ambiguities, then evaluate testability, then evaluate completeness. After each step the tester checks and corrects the result, so the next step works on a clean result.

The other three test analysis tasks are prioritizing test conditions, checking coverage and suggesting test techniques, and each one depends on giving the LLM the right information.

Prioritizing test conditions by risk level

An LLM can recommend priority levels when it is given the risk likelihood and the risk impact of failure for each test condition. Useful risk inputs named by the syllabus include regulatory compliance, user-facing features such as login or payment processing, and historical defect data. The risk data is what the ranking is built on: supply likelihood and impact for each condition, and the LLM can justify each priority from them. A prompt that gives conditions but no risk information can still return a tidy ranking, but nothing in it explains why one condition comes first.

Priority follows likelihood and impact together; neither one alone sets it. Change raises likelihood, a poor defect history raises likelihood, and legal, financial or safety consequences raise impact. Ask the LLM to show both ratings beside each priority so the tester can check the reasoning.

Supporting coverage analysis

By mapping requirements and user stories to test conditions (or test conditions to test cases), an LLM can show whether every aspect of the test basis is covered and list what is not. This is most useful for complex requirements, where gaps lead to escaped defects. Map at the right level. A mapping done only per requirement can mark a requirement as covered when only part of it is tested. Mapping per testable statement (each rule, condition or acceptance criterion inside the requirement) shows partial coverage as well as missing coverage.

Suggesting test techniques

GenAI can suggest test techniques (for example boundary value analysis or equivalence partitioning) based on the type of requirement. The tester still judges whether the suggestion fits, and one requirement may need several techniques:

In test design, GenAI turns test conditions into draft test cases, based on both functional and non-functional requirements. Performance, usability or security requirements are as valid an input as a functional user story; either way the output is a draft for human review. When prompted with suitable information, the LLM can suggest preconditions, inputs, expected results and coverage criteria, producing test cases for different test objectives, from basic functional verification to complex end-to-end testing.

Two prompt problems explain most weak test cases:

• The output format is left to chance. Without one, the LLM decides how much each case contains, and the result may be too thin to execute. The output format is what fixes the shape of every case, so it names the fields an executable case needs.
• The test objective is wrong or implicit. Generated tests aim at the objective the prompt states, and at nothing else. When the purpose is integration or end-to-end testing, the prompt says so and gives the model the interfaces between the components; a prompt that only describes single features gets single-feature cases.

Hands-on objective 2.2.2a shows a typical chain: (1) a structured prompt generates functional test cases from acceptance criteria in a specific output format; (2) a second prompt verifies completeness by having the LLM produce a table mapping each acceptance criterion to the test cases that cover it; (3) a meta prompt helps craft the prompt for end-to-end test procedures. Step 2 turns completeness into something a tester can check row by row, which is the point of asking for a table.

Gherkin test cases with few-shot prompting

Hands-on objective 2.2.2b generates Gherkin (Given-When-Then) test cases with few-shot prompting. The prompt contains several examples, each with a user story, its test conditions and the expected Gherkin scenarios, so the LLM learns both the house style and how a story maps to scenarios. It is then applied to a new user story; if results are inaccurate, the prompt or the examples are refined. Pairing each story with its scenarios is what lets the model learn the mapping, not just the syntax.

GenAI can synthesize test data and generate test procedures and scripts, two implementation tasks that both depend on stating exactly what is needed.

Synthetic test data

GenAI can create representative, data privacy-preserving synthetic test data that resembles production data and covers extreme situations and varied test conditions, for functional and non-functional testing. Its main benefit is that it simulates realistic scenarios without exposing sensitive information. The way to get it is to describe the data: field rules, formats, value ranges and relationships between fields. Describing the data, rather than supplying real records, keeps personal data out of the prompt altogether.

Coverage of extreme situations has to be requested: name each field's limits and ask for values on and around those limits, plus invalid values where negative tests need them. Then review the records against the field rules, because generated data can break them.

Manual procedures and automated scripts

From structured test cases, GenAI can generate both manual test procedures (numbered steps and expected results, suitable for people who do not read code, such as user acceptance testers) and automated test scripts compatible with a test automation framework. Framework compatibility is learned from examples: existing scripts from the target framework show the model its conventions, helpers and structure, so they are the input that matters most after the test cases themselves.

Scripts can also be updated or extended based on new requirements. When a requirement adds or changes behaviour, give the LLM the requirement together with the scripts it touches and ask for targeted updates; the rest of the suite stays as it is. (Self-healing, covered below, is a different capability aimed at minor UI and API changes.)

GenAI can analyse test cases and their interdependencies and propose an optimized test execution schedule based on priority, associated risks, resource availability and test objectives. Priorities and risks alone are not enough: a good schedule also needs the dependencies between test cases (one case creates data another needs) and the availability of testers and test environments.

Hands-on objective 2.2.2c uses prompt chaining to prioritize a test suite under different test prioritization strategies: risk-based, coverage-based and requirements-based. The prompt should name the strategy precisely and keep the dependencies. Each strategy has its own ordering rule (risk level, coverage gained, requirement priority), and in every strategy dependencies remain hard constraints: a schedule that breaks them cannot run.

This is a multi-criteria reasoning task, and LLMs make reasoning errors on such tasks, so the result must be verified manually. In a chain, each step's output is checked before it becomes the next step's input, and an error is corrected where it is found (for example a case planned in an environment that is not available that day). Later steps then build on a sound result instead of inheriting the mistake.

Regression suites grow with every iteration and run often in CI/CD pipelines, which makes them ideal for automation; GenAI helps create, maintain and optimize automated regression suites. Section 2.2.3 names keyword-driven script implementation, impact analysis and test optimization, self-healing and adaptive tests, automated test reporting and insights, and enhanced defect reporting with root cause analysis. They apply to functional and non-functional regression tests, and because GenAI can make mistakes, the output must be checked carefully depending on the associated risk.

Keyword-driven test automation

In a keyword-driven framework, predefined keywords such as "Open Login Page" represent common test steps. An LLM can map keywords to test cases and generate scripts that reuse them. This is a few-shot task: example keyword scripts show the model the expected structure, and documentation of the keyword library (names, parameters, purpose) limits it to keywords that actually exist. Hands-on objective 2.2.3a works through it in two parts. First: document the keyword library, generate initial scripts, have AI validate them, and expand coverage with more scripts. Second: debugging support, using system prompts to create an AI assistant that checks and corrects scripts. In such an assistant, instructions that must hold for every request live in the system prompt, and the material for each request arrives in the user prompt.

Impact analysis and test optimization

GenAI can analyse code changes to identify high-risk areas affected by them, enabling targeted regression testing when the full suite cannot run in time. The selection is a recommendation, and like any GenAI output it is checked in proportion to risk: the more critical the changed area, the more carefully a tester confirms that the selected tests really cover it.

Self-healing and adaptive tests

GenAI can automatically adjust scripts to minor UI or API changes, preventing unnecessary failures and keeping suites stable. It must never absorb a real defect or a changed requirement:

GUI regression tests are often unstable because of recurrent UI changes; GenAI can adapt them to dynamic locators and modified interactions. API regression tests face changing request/response formats, endpoints and authentication; GenAI can adapt scripts to the evolving API specification and generate diverse test data. Expected results change only when a requirement changes; that rule is what keeps self-healing from masking defects.

GenAI turns regression results and logs into reports: automated test reporting and insights (detailed, timely reports with success metrics, failures and key insights, and dashboards showing trends and predictive insights on potential failure points) and enhanced defect reporting and root cause analysis (automatically compiling comprehensive defect reports with test logs, screenshots and test environment data).

Hands-on objective 2.2.3b analyses a regression test report with structured prompts linked in a single LLM conversation, in this order:

1. Compare the test results with the test specification.
2. Cluster similar failures, to see which share a cause.
3. Maintain a known anomalies list and check failures against it, so issues already logged are not reported again as new.
4. Cross-check the findings.

Each step answers a different question: the comparison says what failed against the specification, clustering says which failures share a cause, and the known anomalies list carries memory from run to run, so recurring issues are recognised and only genuinely new ones are reported.

GenAI supports test monitoring and test control by analysing and synthesizing the large, sometimes unstructured data that test management tools already hold. Section 2.2.4 names four tasks, which questions often contrast:

Monitoring compares what is happening with the plan and raises deviations; control decides what to change in response. Prediction depends on trends over time measured against the plan, which is why monitoring prompts include the plan and a history of the metrics, not a single reading. Hands-on objective 2.2.4 demonstrates the flow: test data extracted from test tools, LLM-generated metrics such as test progress, defect trends and coverage that highlight risks, then a dashboard and a natural language summary.

The right prompting technique depends on the characteristics of the test task: complex tasks that need precision and checking use prompt chaining, repetitive or fixed-format tasks use few-shot prompting, and new or flexible tasks start with meta prompting. Section 2.2.5 gives this as a table:

Read the task for its signals. "Same format, many times" points to few-shot. "Nobody has a prompt for this yet" points to meta prompting; few-shot relies on examples, which a brand-new task may not have until a meta prompt helps produce them. "Errors propagate between steps" or "each result must be checked before the next" points to chaining. Human verification in a chain is independent evidence at every step: reviewing intermediate work products, and running generated scripts where that is possible.

Combining techniques

Several techniques can serve one use case. The syllabus example: meta prompting creates an initial prompt; its examples are adapted and enhanced (few-shot prompting); the task is divided into subtasks so intermediate results can be validated (prompt chaining). Each technique answers a specific need (no starting prompt, a format to imitate, steps to verify), so a combination includes the techniques whose needs are present in the context.