CT-GenAI: GenAI Adoption Roadmap, Shadow AI and LLM Selection

A GenAI adoption roadmap is the planned path a test organization follows to bring generative AI into its test processes, and it is built on a GenAI test strategy. The CT-GenAI syllabus says that strategy must consider the test objectives to be achieved, appropriate LLM selection, issues with the input data used for prompting, and compliance with AI standards and regulations. Based on the strategy, the organization establishes a roadmap and monitors progress in integrating GenAI into test processes.

Section 5.1 of the syllabus therefore has four parts, and this lesson follows them in order:

How people learn the skills and how test roles change is the sibling topic, section 5.2 (managing change), and is covered in its own lesson.

Shadow AI is the use of GenAI tools or systems within an organization without formal approval or oversight. The defining feature is the missing approval and oversight. What makes a tool shadow AI is not what kind of tool it is or how good its output is, but whether the organization has approved it and oversees how it is used for work.

The syllabus names three risks:

The three risks are independent of each other. Security and privacy is about what could be exposed; compliance is about which obligation or rule is not met, and it can arise even when no data is exposed; intellectual property is about who owns, or may use, the material and the output, and it can arise even when the tool is well secured. These three are the risks the syllabus attaches to shadow AI specifically; other risks of GenAI are covered in other parts of the syllabus.

The syllabus says that a strategy and steps for integrating and deploying GenAI help test organizations avoid the risk of shadow AI. The reasoning is straightforward: people often turn to unapproved tools when they want GenAI help that the organization does not provide. A strategy sets objectives and rules for GenAI use, and the deployment steps put approved, governed tools in place for real test tasks, with a planned rollout. That gives testers a sanctioned route, so approval and oversight exist from the start.

In this lesson's reading, restrictions and policing can support the strategy, but they do not replace it: the syllabus's answer to shadow AI is the strategy together with the steps for integrating and deploying GenAI.

The introduction to syllabus section 5.1 names four things a GenAI test strategy must consider: the test objectives, appropriate LLM selection, issues with the input data used for prompting, and compliance with AI standards and regulations. Section 5.1.2 then discusses the key aspects in continuous prose, beginning with measurable test objectives. This lesson groups that discussion into five aspects, in the order the syllabus discusses them:

1. Measurable test objectives: what GenAI should achieve in testing, such as increasing test productivity, shortening test cycles and improving test quality.
2. LLM selection aligned with those objectives, while ensuring compatibility with existing test infrastructure and meeting system scalability requirements (the selection criteria are in section 5.1.3).
3. Input data quality: accurate, relevant input data protected by robust security procedures.
4. Training and metrics: comprehensive training for technical and ethical skills, and specific metrics that measure how effective GenAI results are.
5. Process guidelines for compliant, ethical use: rules for sensitive data, transparency obligations and quality gates with review of generated testware.

The order matters. Objectives come first because they are what the model choice is aligned with; choosing an LLM before agreeing what GenAI should achieve leaves nothing to judge the model against.

A good GenAI objective states a test outcome the organization can measure, in terms of the improvements the syllabus names: test productivity, test cycle time and test quality (for example fewer days per regression cycle, or less effort per test case). A figure that only describes how much GenAI is being used is measurable, but it says nothing about whether testing improved, so it is a usage indicator rather than an objective.

Once the objectives exist, the chosen LLM must serve them in practice. Two checks belong with the selection:

• Compatibility with existing test infrastructure: can the model connect to the test management tools, CI/CD pipeline and automation framework that the objectives depend on? A model that cannot be integrated into the test process cannot deliver an objective tied to that process, however well it performs in isolation.
• System scalability requirements: will the solution keep performing as usage grows beyond the first users? Scalability is a requirement to confirm as part of selection and planning, not something to discover later.

Output quality on its own does not make a model the right choice; it must be aligned with the objectives, fit the infrastructure and scale.

The effectiveness of LLM-powered testing depends on accurate, relevant input data, protected by robust security procedures; maintaining high-quality input data is key to results that can be trusted to be correct.

The two halves do different jobs. Quality (accurate, relevant and current) determines whether the output can be correct: an LLM works from what it is given, so errors or outdated content in requirements, test bases or examples carry through into the generated testware. Security protects that data from exposure, but secure data is not automatically accurate data. When generated testware is wrong in a consistent way, it is worth checking what is being supplied to the model, because correcting the input addresses the source, while checks later in the process detect problems rather than prevent them.

The strategy also needs people who can use GenAI well, evidence that it works, and rules for using it responsibly.

• Training: comprehensive training programs so that test teams have both the technical and the ethical skills to use GenAI tools effectively. The syllabus frames this as training for test teams, not for a few individuals. (How to build these skills in practice is section 5.2.)
• Metrics: specific metrics collected to measure the effectiveness of GenAI results, using measures such as those in syllabus section 2.3.1 (accuracy, precision, recall, relevance and contextual fit, diversity, and others). Objectives say what should improve; metrics show whether it did. Other parts of the strategy cannot stand in for this evidence.

The process guidelines exist to ensure compliance with regulatory standards and adherence to ethical guidelines. The syllabus lists three elements:

To select LLMs or small language models (SLMs) for test tasks, the syllabus gives four criteria: model performance, fine-tuning potential, recurring cost, and community and support. Models differ in functional capabilities (for example multimodal input or reasoning), technical features (for example context window size) and licensing types (commercial or open source). Many public benchmarks exist for natural language processing, code generation or image analysis, but only a few focus specifically on software test tasks, which is why careful, organization-specific evaluation is needed.

Because few public benchmarks target test tasks, general rankings say little about how a model will do on a particular organization's testing; an organization-specific benchmark built from its own test tasks and data, scored with metrics, is the evidence the syllabus asks for. Fine-tuning potential has two parts, possible and useful: the model must allow it, and the use case must gain from domain-specific data, which is mainly the case in specialized contexts. The organization may select one or more models that meet its needs and constraints.

The recurring cost of using GenAI for a test task is estimated from the number of input and output tokens, the prompts used and the frequency of the task, applied to each vendor's pricing model. The hands-on exercise compares at least one commercial solution with one open-source licensed model.

For a per-token API, the arithmetic is:

1. Tokens per run × runs per period = total input tokens and total output tokens (calculate them separately).
2. Divide each total by one million and multiply by its own price per million tokens; output tokens are usually priced higher than input tokens.
3. Add the two amounts, then compare the result with the alternative and the budget.

Worked example with made-up prices: a prompt of 2,000 input tokens returns 500 output tokens and runs 1,000 times a month. That is 2,000,000 input tokens and 500,000 output tokens. At $3 per million input tokens and $10 per million output tokens: 2 × $3 = $6, plus 0.5 × $10 = $5, giving $11 a month. Common slips are pricing output at the input rate, forgetting one side, or swapping the rates.

An open-source model has no per-token licence fee, but self-hosting is not free to run: hosting, compute, maintenance and upkeep are operational expenses, and compute grows with usage. Whether self-hosting or a commercial API is cheaper therefore depends on the volume and frequency of the task, so it has to be calculated rather than assumed. Recurring means costs that come back with use or over time (licensing fees and operational expenses), as distinct from one-time project costs.

Adopting GenAI in a test organization involves three key phases: Discovery, Initiation and usage definition, and Utilization and iteration.

So training, model access and first experiments come before ranking use cases; prioritizing use cases and evaluating infrastructure come before full integration. Full integration in the third phase is not the end point: the phase itself consists of continuous monitoring and of measuring and managing the transformation, so that benefits are sustainable and the set-up can scale.

Phases can run in parallel for different use cases. Each use case moves along the roadmap at its own pace, so at any moment an organization can have one use case in routine, monitored use while another is still being explored. This is normal, not a sign of an inconsistent roadmap.

The syllabus also stresses recognizing and addressing early concerns such as fear of job displacement, which can affect adoption and team morale. Such concerns are a people issue, so they are met by acknowledging and discussing them with the team, not by technical or schedule changes; managing this change is covered in section 5.2.