CT-GenAI: LLM-Powered Test Infrastructure, RAG and LLM Agents

An LLM-powered test infrastructure is a system that integrates a large language model into the test process to strengthen automation, reasoning and decision-making. The CT-GenAI syllabus (section 4.1.1) contrasts it with a traditional AI chatbot: a chatbot is built mainly for conversation, while an LLM-powered test tool is built to support testing work itself, by processing test-related queries, analysing requirements, generating test cases and evaluating outputs.

The difference is not the model. Both may call the same LLM. The difference is everything around the model: who is allowed in, which project data is fetched, how the prompt is assembled, and what happens to the answer before anyone sees it. In a chatbot, the tester does all of that by hand, typing or pasting context and copying the answer out. In an LLM-powered test infrastructure, the system does it as part of the test process.

The syllabus treats two kinds of test infrastructure that use LLMs (introduced in section 1.2.2): AI chatbots and LLM-powered test tools. Section 4.1 then describes three ways to build on the second kind:

• The basic architecture: front end, back end, data sources and an integrated LLM (4.1.1).
• Retrieval-Augmented Generation (RAG): feeding the LLM relevant content from your own documents at query time (4.1.2).
• LLM-powered agents: letting the LLM act on other systems through tools, alone or in teams of agents (4.1.3).

A typical LLM-powered test infrastructure has a multi-component design: a front end, a back end, external data sources and an integrated LLM. The design exists so that the interaction with the LLM is secure and efficient, which is why most of the work sits in the back end rather than in the user interface.

A useful rule follows from the table: anything that must hold for every request belongs in the back end, the one component every request passes through. Masking personal data in defect reports before they are placed in a prompt, or attaching the project glossary to each request so the model uses the team's terms, are both back-end jobs: the first is part of data retrieval, the second part of prompt preparation.

The LLM in the architecture may be hosted in one of two ways: as a third-party service accessed through an API, or as a custom in-house model. Either way, the back end is the component that calls it.

The choice is driven by constraints rather than preference: the organization's data-handling rules, the skills and hardware it has, and how quickly it needs a working tool. The table's last row summarizes the usual outcome. Whichever option is chosen, the model sits behind the back end, which holds the access credentials and applies the same authentication, prompt preparation and post-processing to every call.

The back end integrates multiple data sources, and the syllabus names two kinds with different jobs: relational databases for structured data used in testing, such as test cases, and vector databases for semantic retrieval of related content using embeddings.

Real tools often need both: a precise list from the relational store and a meaning-based search over free text from the vector store. Neither replaces the other. The vector database's job is narrow: it holds embeddings of content so related material can be found by meaning. How those embeddings are created and used is the core of RAG (next sections).

An LLM-powered test infrastructure goes beyond a traditional client-server model because it includes intelligent processing components, the LLM and a multi-source back end, rather than a server that only stores and returns data. The syllabus gives four reasons:

1. The LLM is a reasoning component, not just a server. It interprets and reasons over testware such as requirements, code and test results.
2. Insights are generated dynamically from context. Unlike rule-based chatbots that pick scripted responses through predefined matching rules, the system produces test insights from the context it is given.
3. The back end integrates multiple data sources, such as relational and vector databases.
4. The back end post-processes the LLM's output so it aligns with the test process before presentation.

The distinguishing feature is the reasoning component in the middle. A feature that a conventional system can also have, however modern it looks, does not by itself take a design beyond client-server.

Post-processing the LLM's output

Post-processing is the back-end step that enhances the LLM's raw output so that its responses align with the test conditions of the test process before they are presented to the front end. The LLM's answer is a draft, not the final product.

Prompting reduces deviations but cannot guarantee their absence, because the model's output is generated, not computed. Post-processing is where the system inspects the response and makes it fit, for example:

• validating that generated test data respects the field types and value ranges the system under test accepts;
• reshaping a generated defect summary into the fields the defect tracker expects;
• filtering out suggestions that fall outside the current test scope before testers see them.

Post-processing works on the response; prompt preparation shapes the request, and authentication controls who may ask. Of these back-end functions, only post-processing sees what the model actually returned.

Retrieval-Augmented Generation (RAG) enhances an LLM by incorporating additional data sources into its response generation, which increases the relevance and accuracy of its outputs. It combines a retrieval system with a language model: relevant content from your own documents is found at query time and given to the model as context.

RAG matters in testing because an LLM's training data knows nothing about your product. Through RAG, an LLM-powered test infrastructure can draw on the company's enterprise data sources, such as databases, documentation and repositories, and retrieve contextual information in real time. The result is that test analysis and test design stay aligned with the latest specifications, requirements and existing test data.

RAG supplies knowledge at query time and leaves the model itself unchanged. That separates it from fine-tuning (section 4.2), which adjusts the model's parameters on examples and suits domain-specific reasoning, vocabulary and organization-specific output formats; knowledge built in by fine-tuning stays as it was when the tuning was done. RAG also respects the context window: instead of sending a whole document collection, it selects the few chunks relevant to the current query, so updating the index is enough for later answers to use new content.

In short, RAG changes the input the model works from, not the way the model generates.

A RAG system works in two phases: a preprocessing phase that builds a searchable store of your documents in advance, and a user prompt processing phase that uses that store at runtime (inference) for each query.

Preprocessing (done in advance)

1. Chunking. Large documents are broken into smaller chunks, for example 256 to 512 tokens. Small chunks keep retrieval focused on one idea and keep the retrieved text compatible with the model's context window.
2. Cleaning and encoding. Each chunk is cleaned, processed and encoded into a high-dimensional vector (an embedding) using a pre-trained model.
3. Storing. The embeddings are stored, typically in a vector database, so similarity search is efficient at runtime.

User prompt processing (at runtime, per query)

1. Retrieval. The user query is encoded as an embedding, and the system retrieves the chunks from the previously created vector database whose embeddings are most similar in meaning to the query.
2. Generation. The retrieved chunks are fed to the LLM as context. The LLM generates a response that combines its existing knowledge with the newly acquired data, producing a grounded, more accurate answer.

Three consequences follow from these mechanics:

• Matching is by meaning, not keywords. A query about "password expiry" can retrieve a chunk about a "credential rotation policy", because the two embeddings are close in meaning.
• The answer is only as current as the index. Retrieval reads the store built during preprocessing, so the index has to be refreshed when source content changes; until then, answers reflect the earlier text.
• Grounded does not mean "chunks only". The model still uses what it already knows. Project facts (limits, rules, names) come from the retrieved chunks, while general testing knowledge, such as how equivalence partitioning works, comes from the model's training. That blend is what the syllabus means by combining existing knowledge with newly acquired data.

Chunk size is a balance: oversized chunks make retrieval imprecise and spend the context window on unrelated text, while very small chunks can separate a rule from the context that gives it meaning.

RAG's strength is grounding answers in the organization's current content; its limits come from what retrieval returns and how current the index is. The syllabus's hands-on exercise (HO-4.1.2) shows both by comparing the same test task with and without RAG. Without RAG, the LLM answers from its training data and invents or generalizes project details. With RAG, answers quote the right project facts.

A typical limitation appears when an answer depends on several separate pieces of content. Retrieval ranks chunks by similarity and passes on only the best matches, so a relevant piece can be left out and the output is incomplete without being wrong. When diagnosing a RAG answer, ask in order: was the right content in the index, was it retrieved, and did the generation step use it?

An LLM-powered agent is a specialized GenAI application, powered by an LLM, designed for semi-autonomous or autonomous processing of defined tasks. Its core is the LLM's natural language understanding and generation, complemented by the ability to process instructions, retrieve context and take actions.

What separates an agent from a traditional AI chatbot is that it can act: it invokes a predefined set of functions, commonly called tools, which let it interact with and manipulate external systems. A chatbot only answers questions; an agent can, for example, read a test environment's logs, open a defect in the tracker and assign it to a component owner. Tools give the agent reach into other systems.

Agents differ in how much they run without people:

Autonomy is about oversight, not about capability. Both kinds use tools; both can retrieve context.

A multi-agent architecture is a collaborative system in which several agents, each with a specialized role, communicate and coordinate to solve complex problems more efficiently than a single agent. This coordination is called orchestration. In testing, one agent might analyse user stories, another design tests, a third write scripts and a fourth review results, passing work between them and emulating human reasoning and decision-making. The benefit comes from specialization plus coordination: each agent works on a narrower task, and orchestration moves results between them.

Agents in AI assistants: from scripts to goals

In modern testing tools, agents are often exposed to users through AI assistants integrated into the testing workflow. These assistants can turn user stories or requirements into test artefacts and carry out test analysis, design, implementation, execution and reporting in a semi-autonomous manner. The effect is a continuous, end-to-end workflow that reduces manual effort and shortens feedback cycles, and it shifts test automation from script-based execution to goal-driven, agent-based test automation: automation is driven by a stated objective, such as a requirement, rather than by step-by-step scripts written in advance, with testers keeping oversight.

Risks and the two named mitigations

Agents inherit the problems of the LLMs they use: hallucinations, reasoning errors and biases (see section 3.1). An agent can produce incorrect or misleading results, for example a misread log that leads to a wrong verdict, and because it acts automatically, that error can flow straight into a test report and weaken the reliability of the test process. The syllabus names two mitigations:

• Automated verification procedures for the agents' results, for example comparing outputs with an independent oracle before they are used.
• Semi-autonomous agents for critical tasks, so a human checks the output where mistakes matter most.

Both mitigations aim at detecting wrong results: one by automated checking, the other by human review.

Combining agents and RAG

Agents and RAG solve different problems and combine well. Tools let an agent act on other systems; RAG gives it current knowledge from the organization's documents. A workflow that must both carry out actions and reflect the latest project documentation needs both capabilities.