CT-GenAI: LLM Capabilities for Test Tasks, AI Chatbots and LLM-Powered Testing Applications

Large language models bring one core strength to testing: they read and write natural language and code. The CT-GenAI syllabus lists what that adds up to: LLMs process natural language and code, generate coherent text and code, answer questions, summarise information, translate languages and, in a multimodal context, analyse images.

In a test project, that means an LLM can interpret the artefacts testers work with every day (requirements, specifications, screenshots, code, test cases and defect reports) and generate elements of the testware from them. Because those artefacts exist at every stage, LLMs can support activities throughout the whole test process, from analysing the test basis to reporting on completion.

Keep the division of labour clear. The LLM generates: scripts, expected results, summaries, documents. A test tool or a person runs the tests against the system and records the actual results, which the LLM can then be asked to analyse. The model itself does not retain your project; it works from the prompt, the conversation and any context the tool supplies, so the quality of what it produces depends on the test basis you give it.

The syllabus then lists seven key capabilities for test tasks, introduced as some of the key ones (examples, not a closed list), and covered in the next sections. How tokens, context windows and model types work is taught in the lesson on generative AI foundations and key concepts.

The syllabus lists these seven as key LLM capabilities for test tasks. Each one is defined by its input and the output it produces, and that is the most reliable way to tell them apart:

When you need to map a task to a capability, ask two questions: what is the model given, and what kind of artefact comes back? A request that returns questions about a requirement is requirements analysis even if it mentions test cases; a request that returns what the system should do for inputs that are already listed is oracle generation even if it mentions data.

Requirements analysis and improvement means using an LLM to examine requirements and other parts of the test basis for ambiguities, inconsistencies or missing information, and to generate meaningful questions that clarify them in discussions with stakeholders. A requirement such as "the export shall handle large reports without noticeable delay" shows the pattern: the useful output is not a test but questions, such as what size counts as large and what delay is acceptable, and to whom. That makes this capability most valuable early, for example before a refinement meeting and before test design starts.

Test case creation support means using an LLM to suggest test objectives and help generate test cases from system requirements, user stories or any other element of the test basis.

The two capabilities work in sequence. Take a story that says "users are notified of important account changes": which changes are important, on which channel, and how soon? Sound practice is to let the LLM raise those points as questions, settle the answers with stakeholders, and only then generate test cases. The principle is that a test case is only as definite as the requirement behind it: any unclear point the model resolves by itself becomes an unstated assumption inside the tests and their expected results, and nobody has agreed to it.

Making a requirement easier to read is also different from making it testable. Requirements improvement is about removing ambiguity, inconsistency and gaps, not about style.

Test oracle generation means using an LLM to help generate expected results: given a set of test inputs and the rules that govern them, the model states what the system should do or return. Test data generation means using an LLM to produce the inputs: datasets, boundary values and combinations of test data.

These two are easy to confuse because both deal with values. The difference is direction:

Test data generation adds the most when the request asks for what testers would otherwise derive by technique: boundary values (for a username of 3 to 20 characters, lengths such as 2, 3, 20 and 21) and combinations with other inputs (each of those lengths with each account type the form offers). Typical values alone are still test data, but they leave the edges, where defects cluster, untested.

Oracle output is only as good as the rules supplied. Give the model the specification or rule table it must apply; an LLM inventing expected results from general knowledge is guessing, and every generated expected result still needs checking before it is trusted.

Test automation support means using an LLM to generate test scripts from test case descriptions and to improve existing test scripts by suggesting changes and identifying appropriate test techniques.

Two situations show the two halves:

• Generating scripts. A team has written test case descriptions for its order-history API and wants them as automated tests. The LLM's output here is scripts in the team's framework; if the output were new test cases, the capability would be test case creation support.
• Improving scripts. An existing script checks a travel-date picker with one date in the middle of the bookable window. A useful suggestion names a test technique, such as equivalence partitioning and boundary dates at each end of the window, and turns the script into a data-driven one. Judge any suggested change by whether it increases what the test actually covers.

Generated or modified scripts are code like any other: they need review and a run before they join the regression suite.

Test result analysis means using an LLM to analyse test results by creating summaries and classifying anomalies by severity and priority. It is the capability for triage: when a long test cycle across several environments leaves many anomalies, a summary grouped and ranked by severity and priority shows the team where to start.

Testware creation means using an LLM to help create documents such as test plans, test reports (including progress reports) and defect reports, and to keep them updated as the project evolves. If the release date moves and a new mobile platform enters scope, the test manager can use an LLM to update the test plan's schedule and environments and to draft the next progress report on the new basis.

The boundary between the two is the output. Test result analysis condenses many results into a summary and a severity and priority ranking. Testware creation produces or maintains a test document, such as a plan, a report or an individual defect report. Testware here means the work products of the test process, not general project or development documents.

An AI chatbot is a user-friendly, conversational interface that lets testers communicate directly with an LLM in natural language: they type questions, commands or prompts and receive immediate, contextually aware responses.

The defining habit of chatbot use is iterative refinement. If the first answer is too broad, the tester narrows it within the same conversation, for example with prompt chaining (breaking the work into steps and feeding each result into the next; see the lesson on prompting techniques). The conversation keeps its context, so each follow-up builds on what has already been said rather than starting again.

The syllabus describes chatbots as particularly effective for:

• routine tasks and exploratory testing;
• onboarding new testers, by giving quick access to testing knowledge and practices;
• scenarios needing fast feedback, clarification of test concepts, or dynamic exploration of requirements and potential test cases;
• non-technical stakeholders, because the intuitive interface needs no programming, which broadens the user base and encourages adoption.

The trade-off is that a chatbot needs a person in the loop. Someone has to open it, paste in the input and read the answer, so it is a poor fit for work that must run unattended on every build.

An LLM-powered testing application integrates LLM capabilities via APIs to perform well-defined and often automated testing tasks. Instead of a person chatting, a tool, script or pipeline sends the request and uses the response.

Compared with chatbots, these applications offer greater customisation and scalability, so organisations and tool vendors can embed generative AI into existing test frameworks and tools. That makes them the fit for repetitive or complex tasks that must run without anyone starting them, such as test case generation inside a test management tool, defect analysis, or test data synthesis on every commit or build.

The syllabus names a more advanced implementation: organisations can create AI agents designed to perform certain testing roles. Agents are covered in chapter 4 of the syllabus. "More advanced" is not the same as "better for every need": choose the simplest setup that fully meets the requirement.

Choose the interaction model by who starts the task and how often it runs: a chatbot when a person wants to explore, ask and refine; an LLM-powered application when a well-defined task must run repeatedly, at scale or without anyone starting it. The syllabus presents them as complementary, so many teams use both.

Worked reasoning for typical situations:

• A product owner wants to talk through edge cases for an upcoming story and react to ideas as they come: a chatbot, because the work is interactive and needs no programming.
• Every time a requirement changes in the test management tool, the team wants draft test cases attached to it automatically: an LLM-powered application, because no person starts the task and it repeats.
• A team uses a chatbot to draft exploratory test charters and also wants the defect tracker to propose a severity for each new report: both, each where it fits, since the syllabus treats the two models as complementary.

Whichever way testers interact with an LLM, the syllabus states that successful use of generative AI in testing requires strong prompt engineering: carefully designed prompts with clear, specific instructions improve the likelihood that outputs are accurate, relevant and aligned with the testing objectives, and make the support more consistent and reliable.

This applies to integrations just as much as to chat. The prompts inside an LLM-powered application are written by someone, and they are where output quality is decided. If a defect-triage integration returns vague one-line summaries, the lever is the instruction itself: rewrite its prompts to state exactly which content and format each output must have. An instruction that never said what was wanted stays underspecified whatever else changes around it.

Prompt structure and techniques such as prompt chaining, few-shot prompting and meta prompting are taught in chapter 2 of the syllabus. Checking outputs for hallucinations, reasoning errors and bias is chapter 3.