CT-GenAI: Fine-Tuning LLMs and SLMs and LLMOps for Testing
Fine-tuning is the further training of a pre-trained language model on a targeted, labeled dataset so that it performs a specific test task, and LLMOps is the set of practices, tools and processes for developing, deploying, monitoring and maintaining LLMs in production. This lesson covers CT-GenAI syllabus section 4.2: what fine-tuning changes compared with prompting and RAG, when it suits a test task, why a small language model is often enough, the four fine-tuning challenges, the steps of the fine-tuning process, and the LLMOps considerations for adopting GenAI through an AI chatbot, a GenAI test tool or an in-house tool.
On this page10 sections
- What fine-tuning is
- When fine-tuning is the right tool for a test task
- Fine-tuning and RAG together
- Fine-tuning an SLM instead of an LLM
- The four challenges of fine-tuning
- Managing training data quality and overfitting
- The fine-tuning process for a test task
- What LLMOps is
- Three approaches and their LLMOps considerations
- Combining approaches
- Explain fine-tuning as supervised further training that updates a pre-trained model's weights, and distinguish it from prompting, RAG and training from scratch
- Identify test tasks that suit fine-tuning, such as generating test cases in an organization-specific format, and when to combine fine-tuning with RAG
- Explain why fine-tuning an SLM reduces compute needs and which challenges it leaves in place
- Recognize the four fine-tuning challenges (training data quality, overfitting, opacity, compute) from their symptoms and choose a remedy that addresses the right one
- Describe the order of the fine-tuning process and how output assessment leads back to training data quality
- Explain LLMOps and compare the LLMOps considerations for an AI chatbot, a GenAI test tool and in-house development, including how the approaches combine
What fine-tuning is
Fine-tuning is the further training of a pre-trained language model, either a large language model (LLM) or a small language model (SLM), on a targeted dataset so that it performs a specific task or fits a particular domain. The CT-GenAI glossary describes it as a supervised learning process: the model learns from labeled examples, and the training updates the model's weights. The adapted behaviour is therefore part of the model itself and applies to every later request.
That is what separates fine-tuning from the other ways of steering a model that this syllabus teaches:
| Technique | What changes | How long the effect lasts |
|---|---|---|
| Prompting, including few-shot examples | Only the input of one request | One response; it must be repeated in every prompt |
| Retrieval-augmented generation (RAG) | The context added to the prompt at query time, retrieved from data sources | One response; the data stays current because it is fetched when the query runs |
| Fine-tuning | The model's weights, through further training on a labeled, task-specific dataset | Every later response, until the model is trained again |
| Training from scratch | A new model built without a pre-trained starting point | Not what fine-tuning means: fine-tuning always starts from a pre-trained model |
A practical test: if the team only adds examples, templates or documents to what it sends, the model is unchanged. Only training on a dataset changes the weights. A large set of examples placed in prompts is still prompting, and a set of example pairs used for training is not retrieval.
When fine-tuning is the right tool for a test task
Fine-tuning suits needs that should be built into the model's behaviour: the syllabus names giving a generic LLM specialized reasoning for a particular domain and teaching it a vocabulary unique to that field. Training on a targeted dataset lets the model learn domain-specific knowledge and nuances, so its output becomes more accurate and relevant for the intended use.
The syllabus example comes from test design: a model can be fine-tuned to generate test cases from user stories in an output format specific to the organization. The training data is the organization's own user stories paired with their corresponding test cases, so the model aligns with the organization's test process and terminology.
- Signals that point to fine-tuning: prompt templates or few-shot examples do not keep a format consistent across many requests; the output has to use in-house categories, terms or structures that a generic model does not know.
- The data must match the target. Public or generic examples teach a generic format. To learn the organization's format, the dataset must be the organization's own artefacts.
- Learned behaviour lives in the weights. Any model that has not been trained on the organization's examples can only follow the format it is shown in each prompt, whatever its size.
Fine-tuning and RAG together
Fine-tuning and RAG solve different problems, and the syllabus states that the approaches to using GenAI in testing can incorporate both. Fine-tuning shapes how the model behaves (format, terminology, domain reasoning). RAG supplies what the model should know right now by retrieving enterprise data at query time (see the sibling lesson on LLM-powered test infrastructure for how RAG works).
| Need | Better fit | Why |
|---|---|---|
| A stable, organization-specific output format or vocabulary | Fine-tuning | Learned once into the weights; does not depend on each prompt |
| Data that changes often, such as defect reports, recent test results or the current requirements | RAG | Fetched when the query runs, so it stays current without retraining |
Swapping the roles fails in both directions. Putting fast-changing data into a training set means the model is out of date as soon as training ends, and retraining at the pace the data changes is costly. Supplying the format only through retrieved examples is just prompting again, so it inherits the inconsistency that prompted fine-tuning in the first place. When a team needs both a stable format and current data, the combination is fine-tune for the format and retrieve the data.
Fine-tuning an SLM instead of an LLM
Fine-tuning a small language model gives high performance on a specific task without the same computational overhead that fine-tuning an LLM requires. The syllabus describes SLMs as compact models with fewer parameters that are less resource intensive, and notes that a fine-tuned SLM can reach higher performance levels for specific tasks without the overhead an LLM requires. The choice between the two depends on the specific requirements of the task.
- What the SLM eases: the compute challenge. When a single, narrow task needs fine-tuning and the hardware budget is small, a fine-tuned SLM is the efficient option.
- What it does not ease: the other fine-tuning challenges. An SLM still needs high-quality, task-specific training data, can still overfit, and its reasoning is no more transparent than an LLM's.
- What it gives up: breadth. Large models are trained on broad data and keep wider general knowledge; an SLM is built for focused, lightweight tasks.
Two decisions are separate: whether to fine-tune at all (the task needs learned, organization-specific behaviour that prompting has not delivered) and which model to fine-tune (an SLM when compute is the constraint and the task is narrow).
The four challenges of fine-tuning
The syllabus lists four challenges of fine-tuning a GenAI model for software testing. Each has its own symptom, and telling them apart is the core skill for this topic.
| Challenge | What it means | Typical symptom |
|---|---|---|
| Training data quality | Avoid biased or inaccurate results by using high-quality, task-specific training datasets | The model faithfully reproduces errors, outdated behaviour or skewed coverage that were in its training examples |
| Overfitting | The model becomes too specialized to its training data, which harms its performance on new, unseen data; it must keep generalizing across scenarios | Excellent output for inputs like the training examples, poor output for inputs from a new project or product line |
| Opacity | Lack of transparency in how the model makes its decisions or produces its outputs, which complicates debugging and validation | Testers cannot tell why the model produced a given result, so they struggle to check or correct it |
| Computational resources | Fine-tuning needs significant computational resources; the syllabus attaches this challenge to LLMs | Training runs that the available hardware or budget cannot support |
Only the compute challenge is tied to model size. Data quality, overfitting and opacity apply to fine-tuning any language model, so a change that eases one challenge does not automatically ease another: picking an SLM reduces compute but not overfitting, and cleaner data reduces bias but does not make the model's reasoning traceable. The syllabus names no specific remedy for opacity, so treat it as a limitation to plan validation around rather than one that a data or model choice removes.
Managing training data quality and overfitting
Training data quality is managed by building a dataset that is accurate and representative of the task; overfitting is managed by keeping the model general and checking it on data it has not seen. Both are practical dataset decisions.
Data quality: accurate and task-specific
- Remove inaccurate examples. Test cases with wrong expected results, or written for a retired product version, teach the model those errors. A model that learned incorrect examples is working as trained; the fix is the data.
- Cover what the model will meet. If the examples cover only some features, story types or product lines, the output is biased toward them. Coverage gaps in the training set show up as coverage gaps in the output, for example missing boundary-value test cases when few training examples contained them.
- Stay task-specific. Every example should show the task and domain the model is meant to learn; volume that does not reflect them adds nothing the model needs.
Overfitting: keep the model general
- Train on the full range of scenarios the model must serve, not just the main one.
- Evaluate on held-out data, examples kept out of training. This is standard machine learning practice rather than a syllabus term, but it follows directly from the definition: an overfitted model looks excellent on its own training examples, so only unseen data reveals the problem.
- Aim for generalization, not memorization. The goal is a model that handles new inputs well; a model tuned to match its training examples as closely as possible is, by definition, drifting toward overfitting.
Both challenges sit in what the model learned during training, so a remedy has to act on the training data or the training process. Settings applied at inference time or checks applied to the output act after learning has happened.
The fine-tuning process for a test task
The fine-tuning process has four steps, in this order, as the syllabus hands-on objective HO-4.2.1 demonstrates:
- Select an appropriate LLM or SLM as the pre-trained starting point.
- Prepare a dataset tailored to the test task, for example user stories paired with the organization's test cases.
- Fine-tune the model with a suitable solution, such as a machine learning framework.
- Prompt the fine-tuned model and assess the quality of its output.
The syllabus stresses that the demonstration addresses the quality of the training data in particular. The last step is where data problems surface: when the fine-tuned model's output has a gap, the first question is whether the training set contained enough good examples of the missing behaviour. If it did not, the effective response is to improve the dataset and fine-tune again, then reassess: a remedy works when it reaches the cause, and here the cause is in the data.
What LLMOps is
LLMOps (Large Language Model Operations) is the set of practices, tools and processes that streamline the development, deployment and maintenance of LLMs in production environments. The glossary also names monitoring. In short, LLMOps is about running a model reliably over time, not about getting one good answer.
LLMOps questions arise when GenAI moves from a pilot to everyday use: how model updates are released, how output quality is watched over time, who keeps the model and its data current, what it costs, and how data privacy and security are controlled. Prompt engineering, fine-tuning and RAG are techniques that an LLMOps pipeline may contain; they are not the operational discipline itself.
The syllabus places fine-tuning and LLMOps side by side as the two key practices for operationalizing an LLM-powered test infrastructure: fine-tuning adapts the model, LLMOps manages its operational pipeline.
Three approaches and their LLMOps considerations
How an organization adopts GenAI in its test processes determines which LLMOps decisions it must make. The syllabus describes three approaches, each with a different emphasis:
| Approach | Primary LLMOps considerations |
|---|---|
| Using an AI chatbot | Managing data privacy and security risks while optimizing cost. The organization chooses between an LLM-as-a-Service platform, if the necessary assurances are given, and in-house infrastructure with open-source licensed LLMs for greater control. A rigorous assessment of vendor assurances or internal capabilities is critical. |
| Using a test tool with GenAI capabilities | The same concerns as a chatbot (privacy, security, operational cost), plus evaluating the data security and performance assurances offered by the test tool provider. Because such tools typically complement existing test processes, they need a thorough cost-benefit analysis and risk assessment. |
| In-house development of a GenAI-based test tool | Comprehensive control of data privacy and security risks; careful planning of AI resources (computational resources, data storage, staff training); structured processes for validating and maintaining GenAI-specific developments; and the expertise to implement and deploy an LLM-powered test infrastructure. |
What each consideration protects against
- Assessing vendor assurances or internal capabilities protects sensitive test data. The choice between a service and in-house hosting turns on whether the provider can give the assurances needed and whether the organization can run the infrastructure itself.
- Cost-benefit analysis and risk assessment protect the existing test process. A tool that complements current practice has to justify what it adds and what new risks it brings.
- Structured validation and maintenance protect output quality over time. LLMOps covers maintenance and monitoring in production, not just the first deployment, so every change to an in-house tool's model or data needs the same validation as the original release.
- Planning AI resources makes sure the computation, storage and trained staff exist to build and run the tool at all.
For the data privacy and security risks themselves, see the CT-GenAI lesson on GenAI risks (syllabus section 3.2); this topic covers how they shape the operational choice.
Combining approaches
The three approaches are not mutually exclusive. The syllabus notes that an organization might use an AI chatbot for some tasks while developing custom tools for others, so the approaches can run at the same time, depending on the test activities involved. Each one brings its own LLMOps decisions, made for the test activities it serves.
Any of the approaches can also incorporate additional technologies, such as RAG and fine-tuning of LLMs or SLMs, to make GenAI-supported test processes more effective and adaptable. Neither technique belongs to one approach: a commercial tool may use RAG internally and an in-house tool may use a fine-tuned SLM. The syllabus presents both as options an approach may add, not as requirements.
Tip. This topic is examined at the understand level (K2) across two learning objectives, GenAI-4.2.1 and GenAI-4.2.2. The hands-on objective HO-4.2.1 is an observed demonstration and is not itself examinable, but it illustrates the fine-tuning process that GenAI-4.2.1 explains. Expect to distinguish fine-tuning from prompting and RAG by what each changes, decide when fine-tuning (and when fine-tuning combined with RAG) fits a test need, explain the trade-off of fine-tuning an SLM, and match a described symptom to the right one of the four fine-tuning challenges. Scenario items describe a team's results and ask which challenge or which remedy applies, so know what each remedy actually changes. For LLMOps, know its definition and which considerations the syllabus attaches to the chatbot, test-tool and in-house approaches, and that the approaches can be combined.
- Fine-tuning further trains a pre-trained LLM or SLM on a labeled, task-specific dataset and updates its weights; prompting and RAG only change the input of each request.
- Fine-tuning suits domain reasoning, field-specific vocabulary and organization-specific output formats, such as test cases generated from user stories in the company template.
- Use fine-tuning for stable behaviour like format and terminology, and RAG for data that changes often; the two combine well.
- A fine-tuned SLM gives high task performance with far less compute than an LLM, but data quality, overfitting and opacity still apply.
- The four challenges are training data quality (bias and inaccuracy), overfitting, opacity, and compute, which the syllabus ties to LLMs.
- Overfitting shows as strong results on training-like inputs and weak results on new ones; train on all scenarios and evaluate on held-out data.
- The fine-tuning process runs: select an LLM or SLM, prepare a task dataset, fine-tune with an ML framework, then prompt and assess; output gaps usually lead back to the dataset.
- LLMOps covers developing, deploying, monitoring and maintaining LLMs in production.
- Chatbot: privacy, security and cost, with LLM-as-a-Service versus in-house open-source LLMs. Test tool: adds provider assurances and a cost-benefit and risk assessment. In-house: resource planning, structured validation and maintenance, and expertise.
- The approaches are not mutually exclusive and any of them can incorporate RAG or fine-tuning.
Frequently asked questions
What is the difference between fine-tuning and RAG in the CT-GenAI syllabus?
Fine-tuning further trains a pre-trained model on a labeled, task-specific dataset and updates its weights, so the learned behaviour (such as an organization's test case format or terminology) applies to every later request. RAG leaves the model unchanged and retrieves relevant data at query time to add to the prompt, so it suits information that changes often. The syllabus notes that GenAI approaches in testing can incorporate both.
Why fine-tune a small language model instead of a large one?
A small language model is less resource intensive, so fine-tuning it can reach high performance on a specific test task without the computational overhead that fine-tuning an LLM requires. It suits narrow tasks on a limited hardware budget. It does not remove the other fine-tuning challenges: the SLM still needs high-quality training data, can still overfit and is no more transparent.
What are the challenges of fine-tuning a model for software testing?
The CT-GenAI syllabus lists four: avoiding biased or inaccurate results by using high-quality, task-specific training data; mitigating overfitting so the model still generalizes to new, unseen data; addressing the opacity of the model's reasoning, which complicates debugging and validation; and managing the significant computational resources fine-tuning needs, a challenge it ties to LLMs.
How can you tell overfitting apart from a training data problem?
Overfitting shows as a split: the fine-tuned model performs very well on inputs like its training examples and poorly on new, unseen inputs. A training data quality problem shows as the model reproducing errors or gaps that were in its examples, even on familiar inputs, such as wrong expected results or missing boundary tests. Overfitting is checked by evaluating on held-out data; data problems are fixed by correcting and broadening the dataset.
What are the steps of fine-tuning a model for a test task?
Select an appropriate LLM or SLM, prepare a dataset tailored to the test task, fine-tune the model with a solution such as a machine learning framework, then prompt the fine-tuned model and assess the quality of its output. The syllabus emphasizes the quality of the training data, which is usually where output gaps found in the last step lead back to.
What is LLMOps?
LLMOps (Large Language Model Operations) is the set of practices, tools and processes that streamline the development, deployment, monitoring and maintenance of LLMs in production environments. In testing it covers questions such as how model updates are released, how output quality is monitored over time, who maintains the model, and how cost, data privacy and security are managed.
How do the LLMOps considerations differ between an AI chatbot, a GenAI test tool and an in-house tool?
For an AI chatbot the primary considerations are data privacy and security risks and cost, including the choice between an LLM-as-a-Service platform with adequate assurances and in-house open-source LLMs. A GenAI test tool adds evaluating the provider's data security and performance assurances and a cost-benefit analysis and risk assessment. In-house development emphasizes control of privacy and security, planning compute, storage and staff training, structured validation and maintenance processes, and the expertise to deploy the infrastructure.
Does an organization have to choose one GenAI approach for testing?
No. The CT-GenAI syllabus states that the approaches are not mutually exclusive: an organization can use an AI chatbot for some test tasks while developing custom tools for others, implementing them at the same time depending on the test activities. Each approach brings its own LLMOps decisions, and any of them can incorporate RAG or fine-tuning.
Source
This lesson covers the "LLM-Powered Test Infrastructure for Software Testing" domain of the official CT-GenAI exam guide. Vendors revise their guides — check the source for the current version.
Sign up free to mark lessons complete, bookmark topics and track your exam readiness.