CT-GenAI flashcards

Introduction to Generative AI for Software Testing

• Symbolic AI applies rules written by people; classical machine learning needs data preparation, feature selection and training; deep learning learns features itself but still needs people for labelling, tuning and validation; generative AI uses deep learning to create new content.
• The key advantage of generative AI for testing is that pre-trained models apply directly to test tasks with no additional training phase, but the output still needs review.
• LLMs are generative pre-trained transformers that predict the next token at inference; their output is statistically plausible, and plausible is not necessarily correct.
• Tokenization splits text into character, sub-word or word tokens; technical text such as code and logs usually needs more tokens per word than prose, and tokenizers differ between models.
• Embeddings are high-dimensional vectors encoding meaning and usage; tokens with similar meanings sit close together, which enables matching by meaning.
• LLMs are non-deterministic because inference is probabilistic and shaped by hyperparameters; even the lowest temperature does not guarantee identical runs.
• The context window is the amount of preceding text, in tokens, the model can consider; bigger windows hold longer inputs but cost processing time, and content outside the window is not seen.
• Foundation LLMs are general-purpose and need adaptation; instruction-tuned LLMs are fine-tuned on prompt/response pairs; reasoning LLMs add training on multi-step tasks; choose by task complexity.
• Small language models have fewer parameters and suit lightweight, focused tasks.
• Multimodal LLMs turn images into embeddings via vision-language models; VLMs compare visual and textual input, and every reported result must be verified.

• LLMs interpret requirements, specifications, screenshots, code, test cases and defect reports and generate testware, so they support the whole test process.
• The syllabus lists seven key capabilities as examples, not a closed list: requirements analysis and improvement, test case creation support, test oracle generation, test data generation, test automation support, test result analysis, testware creation.
• Identify a capability by what the model is given and what it returns: questions about a requirement, test cases, expected results, inputs, scripts, ranked summaries or documents.
• Test data generation produces inputs (datasets, boundary values, combinations); test oracle generation produces the expected results for given inputs.
• Clarify ambiguous requirements with stakeholders before generating test cases; never let the model silently choose a meaning.
• AI chatbots: direct conversational access, immediate contextual answers, iterative refinement such as prompt chaining; good for exploration, fast feedback, onboarding and non-technical users.
• LLM-powered testing applications: LLM access via APIs for well-defined, often automated tasks, with more customisation and scalability; AI agents for testing roles are the advanced form.
• The two models are complementary, and both depend on strong prompt engineering with clear, specific instructions.

Prompt Engineering for Effective Software Testing

• A structured prompt has six components: role, context, instruction, input data, constraints and output format; combine the structure with prompting techniques.
• Role sets the perspective (tester, test manager, test automation engineer); context describes the test object, its users and the release.
• Instructions are clear, imperative and concise; input data is the material itself (stories, acceptance criteria, screenshots, code, existing tests, output examples).
• Constraints restrict how the instruction is applied to the input data; output format shapes the structure of the response.
• Prompt chaining splits a complex task into prompts and checks each intermediate output before the next step.
• Zero-shot uses no example, one-shot one, few-shot several; examples make output consistent and in line with expectations.
• Meta prompting has the LLM draft or refine prompts that the tester evaluates: efficient, and a form of human-AI pairing.
• Techniques combine; identify each by what is being reviewed between rounds: testing work in chaining, the prompt itself in meta prompting, and examples inside one prompt in few-shot.
• The system prompt is set by the developer or tester, usually hidden, and constant for the session; the user prompt changes with each request.
• The model considers the system prompt and the current user prompt together, so user prompts must still be focused and explicit.

• High-quality, relevant input is the main lever on GenAI output for test tasks: give the data the task depends on.
• Test analysis with GenAI produces test conditions such as acceptance criteria, findings on the test basis, risk-based priorities, coverage mappings and technique suggestions.
• Risk prioritization needs likelihood and impact per condition (regulation, user-facing features, defect history); without risk data the ranking is only plausible.
• Coverage gaps show up when the test basis is mapped to test conditions or test cases, at the level of each testable rule.
• Ask for preconditions, inputs and expected results in the output format, and state the real test objective, to get executable test cases.
• Synthetic test data is realistic and privacy-preserving; describe the field rules and ask explicitly for values around each limit.
• Keyword-driven scripts come from few-shot examples plus the keyword library documentation; framework scripts follow example scripts from that framework.
• Self-healing adapts to minor UI or API changes only; changed behaviour means updating scripts from the requirement, and incorrect behaviour is a defect.
• Regression report analysis: compare with the specification, cluster failures, check the known anomalies list, cross-check.
• Chaining for complex tasks checked step by step, few-shot for repetitive or fixed-format tasks, meta prompting for new tasks; combine them when several needs apply.

• Accuracy is overall correctness against expert-written tests, requirements or other standards; precision is correctness for one objective (of what was flagged, how much is real); recall is finding all relevant instances (such as every valid and invalid partition).
• Relevance and contextual fit asks whether output is consistent with the test basis and the domain's requirements; diversity asks whether it covers varied user behaviours and edge cases without repetition.
• Execution success rate counts generated artefacts that run as is, without syntax or format errors; it says nothing about whether they check the right results.
• Time efficiency compares AI generation time with the manual effort for equivalent tests; it is the one metric that does not judge output quality.
• Task-specific metrics can be tailored to how well GenAI supports a particular test activity, on top of the general metrics.
• Metrics are evaluated by manual review or automatically, for example against a predefined reference; volume and the existence of a reference decide which.
• GenAI is non-deterministic, so metrics must rest on statistically relevant data: one run proves nothing, and compared versions need comparable runs on the same inputs.
• The five refinement techniques are iterative prompt modification, A/B testing of prompts, output analysis, integrating user feedback, and adjusting prompt length and specificity.
• Sharing practices and prompt libraries across teams standardises techniques, keeps quality consistent and helps teams avoid repeating errors.

Managing Risks of Generative AI in Software Testing

• Hallucination: output that appears factually incorrect or irrelevant to the task, such as fictitious test cases, non-functioning scripts or tests for non-existent acceptance criteria
• Reasoning error: misreading cause and effect, conditional logic or step-by-step processes, because LLMs pattern-match rather than reason; test planning and prioritization are prone
• Bias comes from the training data and shows as a skew across the output, for example in generated test data, refined acceptance criteria or the mix of test types
• Detect hallucinations by cross-verification, domain expert consultation and consistency checks; reasoning errors by logical validation and output testing; bias by reviewing output against the test strategy and coverage
• How much detection to apply depends on the estimated risk of these defects in the test task
• Mitigate with complete context, verified prompt chains, clear data formats, a model trained for the task and comparison across models; disagreement between models flags an item to verify against the test basis
• Non-determinism means a defect fixed in one output can return in another conversation, so outputs need checking every time
• Lower temperature and a fixed seed reduce variability but cannot guarantee reproducibility, especially for long outputs; automate output verification for consistent evaluation

• Three privacy concerns: unintentional data exposure (sensitive data in the output), lack of control over data usage (stored or processed without consent or control) and compliance risks (legal disputes from breaking rules such as the GDPR)
• Security risks: attacks on LLM-powered test infrastructure (data breaches, unauthorized access), exploitation of LLM vulnerabilities, and malicious input that misleads the model
• Context manipulation extracts training data (for example by overflowing the context window); request manipulation disrupts output (for example misleading images); data poisoning manipulates training data (for example fake report ratings); malicious code generation plants backdoors (for example a channel to a malicious IP)
• Core measures: data minimization, anonymization and pseudonymization, secure storage and transmission (encryption, access control), and training programs and policies
• Additional strategies: systematic human review of output, comparison with another LLM, a secure operational environment, regular audits and vulnerability assessments, and staying updated with security best practices
• Minimize first; when a sensitive field is itself under test, use invented values with the same structure; a description of formats and rules often replaces real records entirely
• Environment by confidentiality: commercial secure offering, secure cloud, or in-house installation, and none of them removes the need for other controls
• The GDPR does not restrict GenAI explicitly, but its lawfulness and purpose-limitation safeguards still apply; involve senior security engineers, legal counsel, the CTO or the CISO
• The strategies are complementary and a combination is required

• Training and using LLMs needs many specialized computing resources; web-based use loads devices, networks and data centres.
• Task complexity and required computational resources drive the energy of a single task: one image with a powerful model can use about a smartphone charge, text a small percentage of one.
• Energy rises sharply with usage: one request looks negligible, but the cumulative effect across many requests and users is substantial.
• Accurate energy data is hard to obtain, yet GenAI clearly contributes significant CO₂ emissions; few providers publish per-prompt figures and methods differ, so teams estimate (for example with simulators).
• Limit unnecessary model interactions (reuse approved results, trigger generation by change, match extra checks to risk) and select right-sized models per task.
• Balance the benefits of GenAI automation against its cost and energy consumption, task by task.
• ISO/IEC 42001 (standard): AI management system requirements; ISO/IEC 23053 (standard): ML lifecycle framework for data quality, transparency and safety.
• EU AI Act (regulation, binding): classifies AI by risk and mandates transparency, accountability and bias mitigation; NIST AI RMF (US framework, voluntary): fairness, transparency, security.
• The four guidelines are examples; organizations must track evolving regulations, standards, national laws and frameworks as their context changes.

LLM-Powered Test Infrastructure for Software Testing

• An LLM-powered test infrastructure integrates an LLM into the test process; a chatbot is mainly conversational.
• The back end handles authentication, data retrieval, prompt preparation, interaction with the LLM and post-processing; the front end is only the interface.
• The LLM can be a third-party service via API or a custom in-house model; confidentiality, skills and hardware decide which fits.
• Relational databases serve structured test data with exact fields; vector databases find content with similar meaning through embeddings.
• RAG adds retrieved content from your own data sources to each prompt, increasing relevance and accuracy without retraining the model.
• Preprocessing chunks documents (for example 256 to 512 tokens), encodes them as embeddings and stores them; retrieval then matches the query by semantic similarity against that previously built store.
• In generation, the LLM combines its existing knowledge with the retrieved chunks; answers are only as current as the index.
• Agents act on external systems by invoking tools; autonomous agents need minimal human intervention, semi-autonomous ones have periodic oversight.
• Orchestration coordinates specialized agents; AI assistants shift automation from script-based execution to goal-driven, agent-based automation.
• Agents can hallucinate, reason wrongly or show bias; mitigate with automated verification of results and semi-autonomous agents for critical tasks.

• Fine-tuning further trains a pre-trained LLM or SLM on a labeled, task-specific dataset and updates its weights; prompting and RAG only change the input of each request.
• Fine-tuning suits domain reasoning, field-specific vocabulary and organization-specific output formats, such as test cases generated from user stories in the company template.
• Use fine-tuning for stable behaviour like format and terminology, and RAG for data that changes often; the two combine well.
• A fine-tuned SLM gives high task performance with far less compute than an LLM, but data quality, overfitting and opacity still apply.
• The four challenges are training data quality (bias and inaccuracy), overfitting, opacity, and compute, which the syllabus ties to LLMs.
• Overfitting shows as strong results on training-like inputs and weak results on new ones; train on all scenarios and evaluate on held-out data.
• The fine-tuning process runs: select an LLM or SLM, prepare a task dataset, fine-tune with an ML framework, then prompt and assess; output gaps usually lead back to the dataset.
• LLMOps covers developing, deploying, monitoring and maintaining LLMs in production.
• Chatbot: privacy, security and cost, with LLM-as-a-Service versus in-house open-source LLMs. Test tool: adds provider assurances and a cost-benefit and risk assessment. In-house: resource planning, structured validation and maintenance, and expertise.
• The approaches are not mutually exclusive and any of them can incorporate RAG or fine-tuning.

Deploying and Integrating Generative AI in Test Organizations

• Shadow AI is the use of GenAI tools or systems in an organization without formal approval or oversight.
• Shadow AI risks: data breaches from weakly secured personal tools, non-compliance with standards and regulations (legal consequences), and IP disputes from unclear licensing.
• A strategy and steps for integrating and deploying GenAI, with approved tools for real test tasks, is how the syllabus says to avoid shadow AI.
• A GenAI test strategy starts with measurable test objectives such as productivity, shorter test cycles and better quality; the LLM is chosen to fit them.
• The chosen LLM must also be compatible with the existing test infrastructure and meet scalability requirements.
• Trustworthy results need accurate, relevant input data protected by robust security; training must cover technical and ethical skills, and metrics must measure the effectiveness of GenAI results.
• Process guidelines cover rules for sensitive data, transparency about what was generated, and quality gates with review of generated testware.
• LLM/SLM selection criteria: performance on the organization's own benchmarks, fine-tuning potential, recurring cost (licences plus operations), and community and support.
• Recurring cost = input and output tokens × price × frequency; self-hosted open-source models avoid per-token fees but still have operating costs.
• Adoption phases: Discovery, Initiation and usage definition, Utilization and iteration; they can run in parallel per use case, and early fears of job displacement must be addressed.

• GenAI adoption needs structured change management with two sides: technical skills and organizational aspects (evolved roles and processes).
• The three core skills are prompt engineering, understanding model context windows and developing test review methods.
• Testers combine domain and test expertise with AI skills to evaluate LLM-driven test case generation, defect report analysis and test data generation.
• Key competencies: assess LLM capabilities, refine prompts, evaluate AI-generated testware; essential knowledge: GenAI's inherent risks plus common mitigations.
• Data sanitization removes or masks sensitive, personal or confidential information before it is shared with an LLM.
• Right-sized models and efficient usage patterns balance automation benefits against cost and energy; they do not replace data protection.
• Capability is built hands-on: various LLMs/SLMs, structured learning paths, guided exercises, peer learning and gradual integration into daily tasks.
• A prompt pattern is a reusable prompt template for consistent, reliable output; communities of practice share pattern libraries and lessons learned.
• Testers become AI-assisted test specialists who review AI output, refine prompts and maintain test-specific prompt libraries.
• Test managers add AI-based test strategy, AI-based risk management and monitoring and control, and oversee hybrid teams of people and GenAI agents.

Yes — the whole CT-GenAI deck is free and needs no account, as is every CT-GenAI revision lesson. Create a free account to save progress and drill practice questions; full-length mock exams are a one-time unlock per certification.