CT-GenAI: Evaluating GenAI Test Results and Refining Prompts

GenAI output for a test task needs measuring because an LLM can return test cases that look complete and well written while being wrong, incomplete, repetitive or unusable, and only an explicit evaluation shows which. The CT-GenAI syllabus says that evaluating GenAI in software testing requires a clear set of metrics to assess the quality, relevance and effectiveness of the generated outputs, and that these metrics, whether general or task-specific, help optimize LLM prompting.

Section 2.3 has two halves that depend on each other:

• Metrics (2.3.1) tell you how good the output is, and in which way it falls short: correctness, completeness, fit to the context, variety, executability and effort saved.
• Evaluation and refinement techniques (2.3.2) use those readings to improve the prompt: modify it step by step, compare versions, analyse errors, listen to the testers who use the output, and tune the prompt's length and detail.

The link between them is the point of the topic. A refinement is only an improvement if a metric shows it, and a metric is only useful if it points to something in the prompt you can change. The syllabus hands-on objectives make the same connection: first observe task-adapted metrics applied to real LLM results (HO-2.3.1), then start from an initial prompt and refine it using techniques such as A/B testing and human verification (HO-2.3.2).

The syllabus lists seven general metrics for evaluating the quality and efficiency of GenAI results on a test task. Each one asks a different question about the output, and each comes with a typical testing example.

A useful way to group them: accuracy, precision, recall and relevance judge whether the content is right; diversity judges whether it is varied; execution success rate judges whether it is usable without repair; time efficiency judges whether it was worth generating at all. Time efficiency is the only one that says nothing about the quality of the output itself.

Accuracy, precision and recall all deal with correctness, but each looks from a different angle: accuracy is correctness overall, precision is correctness of what was produced for one objective, and recall is completeness against everything that should have been found.

• Accuracy compares the output as a whole with a trusted standard, such as the requirements or a reference suite written by domain experts. Ask: does the generated material agree with what experts or the requirements say is correct?
• Precision narrows that to a single objective, typically finding anomalies. It looks only at what the output claims. Ask: of the things it flagged, how many are real? Many false alarms mean low precision, however many real defects are among them.
• Recall looks at what the output should have caught. Ask: of all the relevant instances, how many did it find? Missing partitions, missing defects or missing cases mean low recall, however correct the produced cases are.

A worked example with made-up numbers: a reference data set contains 20 known defects. Generated tests flag 25 items, of which 15 are real defects.

The two readings can move independently, so a prompt is judged on both rather than on whichever number looks better. To calculate precision you need only the flagged items and a check of each; to calculate recall you also need to know the full set of relevant instances, which is why seeded defects or a defined set of partitions are used.

No single metric describes GenAI test output fully: a set of tests can score high on one metric and low on another, so evaluations combine metrics to see where the output actually falls short. The pairs below are the ones most easily confused.

When an evaluation finds one strength and one weakness in the same output, name each with its own metric. A good score in one area does not make up for a poor score in another, and the weak metric is the one that tells you what to change in the prompt.

Beyond the seven general metrics, the syllabus says task-specific metrics can be tailored to evaluate how well GenAI supports a particular test activity. The general list is a starting point; a team generating keyword-driven scripts, analysing test reports or prioritising regression tests can define measures that reflect what that activity needs. A task-specific metric is tied to the test activity and its goal. For regression test prioritisation, for example, a team might measure how many of the historically failure-prone areas the model places in the first part of the run; for test report analysis, how many failures it groups under the right root cause.

The syllabus gives two ways to evaluate metrics, and they are often combined:

The deciding questions are volume and whether a reference exists. A team that generates thousands of test data records each release can validate them against a schema or a reference data set by tool, which no reviewer could match at that scale. Whether a generated scenario respects an insurer's long-standing underwriting practice is a different matter: if that practice is not documented, a tool has nothing to compare against and a specialist has to judge it. Writing such knowledge down as a reference is what later makes automated evaluation possible.

GenAI metrics must rest on statistically relevant data, because an LLM is non-deterministic: the same prompt with the same input can produce different output on different runs. The syllabus states this directly: given the non-deterministic nature of GenAI, the metrics must be based on statistically relevant data.

In practice this means:

• One good run is not evidence. A refined prompt that produced better tests once may simply have been lucky; run it enough times to see its typical result before declaring it an improvement.
• Compare like with like. A fair comparison of prompt versions holds everything except the prompt constant: the same test basis, the same model and settings, and a similar amount of evidence for each version. Otherwise the comparison measures luck or input differences rather than the prompt.
• Look at spread, not just averages. A prompt that is excellent sometimes and poor sometimes may be less useful than one that is reliably good, because the tester cannot tell in advance which run they will get.

This rule applies to every metric and every technique in this lesson. It is what turns A/B testing of prompts from an impression into a measurement.

The syllabus names five techniques that build on the metrics to evaluate prompts and improve AI results. Each starts from a different source of evidence.

The techniques are not alternatives to choose between once. A team typically analyses the output to find error types, modifies the prompt in response, A/B tests the new version against the old one over enough runs, and checks with testers whether the result is actually more useful.

The right refinement depends on what the evaluation found, and the near-twin techniques are separated by their evidence and their method.

• Iterative modification vs A/B testing. Iterative modification refines one prompt in sequence, each change prompted by the last result. A/B testing compares several versions side by side on agreed metrics. A tester who keeps editing one prompt after each review is iterating; a tester who writes three phrasings and scores them is A/B testing.
• Iterative modification vs adjusting length. What defines the technique is why the prompt changes. Iterative modification responds to the last results with whatever change they call for, which may make the prompt longer. Adjusting length and specificity is a deliberate experiment with how much detail to give, in either direction, with the versions compared afterwards.
• When shorter may be better. The syllabus says only that more context sometimes helps and shorter prompts sometimes generalise better. One way this can show up: a prompt packed with detailed constraints for one feature may steer every output towards those constraints, and a leaner version may give a broader spread. Neither direction is assumed; the versions are compared over enough runs.
• Output analysis vs user feedback. Output analysis checks the content against the test basis and classifies the errors (for example a business rule stated wrongly). User feedback reports whether the output is useful to the people who work with it, for example whether its level of detail and wording suit them. Error types found by analysis usually lead to correcting or adding the context the model got wrong; feedback on usefulness usually leads to changing what the prompt asks for, such as the format, granularity or audience.
• Metrics and feedback together. The agreed metrics show whether a version is better on measurable properties; testers' views on clarity and usefulness add evidence that those metrics do not capture. Integrating user feedback is one of the five techniques, so it counts as evidence alongside the scores.

All five techniques change the prompt and then measure the effect. Prompt chaining, few-shot and meta prompting are prompting techniques from section 2.1, covered in the lesson on prompt structure and techniques.

Test teams can use the five techniques to organise prompt evaluation and optimization sessions that keep improving their GenAI prompts, and share what they learn across the team or the whole test organization. The syllabus says that sharing practices:

• helps standardise prompt techniques and maintain consistent quality between teams and projects;
• promotes a culture of learning and iterative improvement;
• lets teams build on collective insights, avoid repeated errors and refine their use of GenAI tools over time, for example by sharing prompt libraries.

In practice a session might take one recurring task, such as drafting acceptance tests from user stories, settle what will count as a better result before anyone runs a prompt, try the candidate prompts with repeated runs, and then record the winning prompts, with their measured results and what was learned from the weaker ones, where other teams can find them. Because models, products and needs change, library entries are revisited too: continuous improvement applies to the library as much as to any single prompt.