CT-GenAI: Generative AI and LLM Fundamentals for Software Testing

The CT-GenAI syllabus places generative AI at the end of a spectrum of four AI approaches, and each one differs in where its decision logic comes from. Symbolic AI is given its logic by people; classical machine learning and deep learning learn it from data during training; generative AI reuses large models that were already pre-trained and creates new content from the patterns they learned.

Models such as LLMs can generate text, write code and simulate reasoning or problem-solving within the scope of their training: what they produce is bounded by the patterns they learned.

Two relationships are worth fixing in memory. First, generative AI is built on deep learning: it uses deep learning techniques, not the other way round. Second, deep learning removes manual feature definition, not people: a deep learning project still needs labelled examples, tuning and validation.

The syllabus mentions other AI technologies exist but leaves them out of scope; this spectrum of four is what you need.

Each approach has a characteristic strength and a characteristic cost, and a scenario usually turns on one of them: how explainable the decisions must be, what shape the data has, whether labelled data exists, and how much time there is for training.

Read a scenario for its binding constraint and pick the approach whose strength meets it. Learning from a project's own data is the strength of the training-based approaches; inspectable, rule-by-rule decisions are the strength of symbolic AI.

The key advantage of generative AI for testing, in the syllabus's words, is that it uses pre-trained models that can be applied directly to test tasks without an additional training phase. That advantage comes with risks (hallucinations, reasoning errors and biases, covered in chapter 3): skipping training does not make the output correct, so generated test artefacts still need review.

Generative AI (GenAI) is the branch of AI that uses large, pre-trained models to generate human-like output such as text, images or code. A large language model (LLM) is a GenAI model pre-trained on very large textual datasets (books, articles, websites) so that it can determine context and produce relevant responses to a user's prompt.

LLMs are based on the generative pre-trained transformer, a deep learning model. The transformer is a neural network architecture that processes the context of long text sequences and learns how tokens relate to each other. At inference time (when the model answers a prompt) an LLM works one step at a time: it predicts the next token in the sequence from the relationships it learned in training, appends it and predicts again. Its parameters were fixed by training; answering a prompt uses them without changing them.

Small language models (SLMs) are compact models with fewer parameters than LLMs, designed to give lightweight and focused GenAI solutions. Size is what defines them. A smaller model is cheaper and faster to run and easier to host on the team's own hardware, which helps when data must stay in-house; the trade-off is capability, so tasks with long inputs or several dependent steps of logic favour larger models.

Plausible is not necessarily correct

Because an LLM generates the next token that is statistically plausible given its training data and the prompt, its output can be fluent, well formatted and confident while still being wrong. The syllabus states it directly: plausible is not necessarily correct.

It helps to separate three sources of wrong output. Randomness in sampling shows up as answers that change between runs. Lost input shows up when the material is too long for the model to consider in full. Plausibility-driven generation can produce an error that is consistent: the most likely-looking continuation is simply wrong, run after run, even with a short and complete prompt.

The practical rule: fluency, formatting and repeatability are no evidence of correctness. Reducing randomness makes answers more consistent, not more accurate, so LLM output cannot stand in for an unchecked test oracle. Check generated expected results, test data and test cases against the test basis. Chapter 3 names the failure types (hallucinations, reasoning errors, biases) and how to mitigate them.

Tokenization is the process of breaking text into smaller units called tokens, which can be as small as a single character or as large as a sub-word or a whole word. An LLM tokenizes its input first, so that each token can be processed individually while the overall context is kept.

Token count, not word or character count, is what an LLM measures. Common words in ordinary prose often map to a single token, while rare strings are split into several sub-word or character tokens. Technical test material (source code, identifiers, encoded or machine-generated values, paths, symbols) is full of rare strings, so it usually needs noticeably more tokens per word than prose does.

• Tokenizers differ between models. The same text can give different token counts on different models, so a count taken with one model's tokenizer is not reliable for another, even when the advertised window sizes match. Measure with the tokenizer of the model you will use.
• Character-based rules of thumb are rough. They are calibrated on ordinary prose and tend to underestimate technical text. When an input is anywhere near the limit, measure it rather than estimate it.
• Tokenization comes before embeddings. Text is first split into tokens; each token is then turned into an embedding vector. The two are steps of one pipeline.

Hands-on HO-1.1.2: tokenization and token count evaluation

The syllabus's hands-on exercise has two parts. Tokenization: run a tokenizer on sample text and examine how words, punctuation and phrases are represented. Token count evaluation: measure the tokens produced by different inputs (a user story, a test log, source code) and analyse how token count affects the model's behaviour against its context window limit and its efficiency. The aim is to anticipate how text structure and input length will affect an LLM interaction before you send it.

Embeddings are numerical representations of tokens: each token is turned into a vector in a high-dimensional space that encodes its semantic, syntactic and contextual relationships, that is, its meaning and how it is used. Tokens with similar meanings or contextual roles have embeddings that sit close together in that space. Each token keeps its own vector; similarity is expressed as distance between vectors.

That proximity is what lets an LLM understand word relationships, keep context and respond coherently. It is also why LLM-based tools can relate items by meaning rather than by matching characters: a test case about "deleting an attachment" can be related to a defect report about "removing a file" because the meanings are close in embedding space, even though the wording differs.

Tokenization and embeddings together convert language into a numerical form the model can process. Embeddings are learned during training and are part of the model.

LLMs are non-deterministic: the same input can produce different outputs. The syllabus gives two causes: the probabilistic nature of inference (each next token is chosen from a probability distribution) and hyperparameter settings such as temperature that control how widely the model samples.

So the variation comes from the generation step itself. The parts of the pipeline fixed by training and design behave the same on every request; it is the sampling of each next token that differs.

Lowering the temperature narrows the sampling and reduces variation, but providers do not promise identical output on every run even at the lowest setting. For testing this has two consequences:

• Any automated check that expects LLM output to repeat exactly will be flaky. Build checks that tolerate variation: validate structure, required content and rules rather than exact text.
• Agreement between runs shows consistency, not correctness. Results still need checking against the test basis.

The context window is the amount of preceding text, measured in tokens, that an LLM can consider when generating a response. Anything outside it is simply not seen by the model.

A larger context window lets the model stay coherent over longer passages, such as a large test log. The cost is that more tokens in the window increase the computational complexity and the processing time. Window size is a capacity: it decides how much fits, and a fuller window means slower, more expensive requests.

Context problems are capacity problems, so the fix is to change what goes into the window. Variation between identical runs is a different issue (see non-determinism above).

LLMs are built through progressively specialised training stages, giving three categories: foundation, instruction-tuned and reasoning LLMs. Each later stage starts from the one before it.

A foundation model has learned to predict text, not to follow requests, so a prompt may simply be treated as the start of a document to be continued. Fine-tuning on prompt/response pairs is the stage that aligns a model to act on instructions. Better instruction-following is not the same as stronger multi-step reasoning: that comes from the further, reasoning-focused training stage.

The syllabus notes that both instruction-tuned (sometimes called non-reasoning) and reasoning LLMs are used in testing. The choice depends on the complexity and reasoning demands of the task. Because reasoning models usually cost more and respond more slowly, match capability to need: use an instruction-tuned model where the task is simple and reserve the reasoning model for work that genuinely needs multi-step reasoning.

A multimodal LLM extends the transformer model to process several data modalities: text, images, sound and video. It is trained on large, diverse datasets so it learns relationships between the data types. Tokenization is adapted for each modality: images are converted into embeddings by vision-language models, and those embeddings are processed in the transformer together with the text tokens. Image and text end up as vectors the same model works on together.

Vision-language models (VLMs) are a subset of multimodal LLMs that specifically integrate visual and textual information, for tasks such as image captioning, visual question answering and analysing the consistency between textual and visual input. In testing, LLMs augmented with VLMs can analyse screenshots and GUI wireframes alongside user stories or defect reports, find discrepancies between expected results and what is actually on screen, and generate test cases that use both textual data and visual cues, increasing coverage. Checking visual evidence needs a model that receives the image itself.

Images also consume the model's capacity. The more visual input a single prompt carries, the less thoroughly each part tends to be analysed. Keep each prompt's visual input to what the model can examine properly, pair each image with the text it should be checked against, and check each result.

A good multimodal prompt for a test task names the inputs, states the comparison to make and asks for output that can be checked item by item. Hands-on objective HO-1.1.4 has two steps: review the inputs (the given prompt, the text and the image) and execute the prompt and verify the actual result (submit both the image and the text to a multimodal LLM and check its response).

For example, with a wireframe of an account-settings page and a user story's acceptance criteria, an effective instruction might read: "For each acceptance criterion below, state whether the attached wireframe satisfies it, describe the element you relied on, and say so if the criterion cannot be assessed from the image." It keeps the model on the comparison, uses both inputs, and makes every claim traceable to something you can look at. A prompt that leaves the task, one of the inputs or the output shape unstated invites the model to do something other than the comparison you need.

Verification is part of the task, not an option. A response that reports everything as satisfied may be right, or it may have missed or invented details. Every reported result is checked against the image and the test basis before it is recorded; whatever was not checked remains unverified. The exercise is meant to show both the benefit (fast visual-textual comparison) and the challenges (missed elements, plausible but wrong verdicts).

This lesson covers the foundations. The next topic of the syllabus (section 1.2) covers the key LLM capabilities for test tasks and the choice between AI chatbots and LLM-powered testing applications; chapter 2 covers prompt structure and prompting techniques in depth.