CT-GenAI: Data Privacy and Security Risks of GenAI in Testing

Using Generative AI in software testing creates data privacy and security risks for two reasons the CT-GenAI syllabus names: test tasks feed the model sensitive information (requirements, defect logs, production-derived test data, source code), and the LLM-powered test infrastructure itself has vulnerabilities that attackers can exploit. Robust data protection is therefore needed to prevent breaches, unauthorized access and exposure of confidential data.

The syllabus splits the subject into three parts, and this lesson follows the same order:

A hands-on objective (H0) adds a demonstration: recognising these risks in a given case study of GenAI used for testing. Hallucinations and bias are covered in the sibling lesson on section 3.1, and named AI regulations and standards beyond GDPR in the lesson on sections 3.3 and 3.4.

GenAI can process large amounts of data that may contain sensitive or personally identifiable information, and the syllabus names three data privacy concerns that follow from this: unintentional data exposure, lack of control over data usage, and compliance risks.

The quickest way to tell the first two apart is where the data ends up. Unintentional exposure is about what comes out of the model: sensitive content surfacing in a response. Lack of control is about what happens to data that went in: it is stored, retained or passed on in ways the submitter never agreed to or cannot govern. Compliance risk is about the legal consequence of processing personal data the wrong way, whichever tool is used.

Beyond privacy, the syllabus names three specific security risks when testing with GenAI: the infrastructure can be attacked, LLM vulnerabilities can be exploited, and malicious input can be introduced on purpose.

• Attacks on the test infrastructure. LLM-powered test infrastructure can be vulnerable to security attacks such as data breaches or unauthorized access. A server holding prompt history, generated test data or model logs is an attractive target in its own right: if a misconfigured storage bucket leaves generated test datasets readable from the internet, that is an infrastructure weakness, not a flaw in the model's output.
• Exploiting LLM vulnerabilities. Malicious actors can exploit vulnerabilities in LLMs, such as the manipulative attacks listed in the next section, to alter the model's behaviour or extract sensitive information.
• Malicious input data. Attackers can intentionally introduce malicious input to mislead the LLM and compromise its accuracy or security. This risk grows whenever a tool passes content it does not control straight into a prompt, for example app-store reviews, customer support tickets or web pages the tool scrapes. Hidden or crafted text in such input can steer the model toward output that serves the attacker rather than the tester.

Note the difference from the privacy concerns: these risks involve an attacker or an attack. They are also distinct from the quality defects covered in section 3.1 (hallucinations, reasoning errors, bias), which occur without anyone attacking the system.

The syllabus gives four example attack vectors against GenAI test processes and tools: context manipulation, request manipulation, data poisoning and malicious code generation. Each has its own goal and its own syllabus example, and exam items commonly describe the example and ask for the vector.

Telling the near-twins apart

• Context manipulation vs unintentional data exposure. Both end with sensitive data in the output. Context manipulation is a deliberate request crafted to extract it; unintentional exposure happens without anyone intending it.
• Request manipulation vs data poisoning. Request manipulation disrupts the output through what is sent in a request now; data poisoning alters the training data, so the effect shows up after the model learns from it. The useful question is whether the attacker's data is used once, in one response, or kept and learned from.
• Request manipulation vs context manipulation. Request manipulation wants a wrong output (for example invented acceptance criteria); context manipulation wants confidential output (training data).
• Malicious code generation is recognised by what the generated code does rather than how well it tests: for example an undocumented user account created in a setup step, credentials written to a location outside the test, disabled certificate validation, or obfuscated strings with no testing purpose. It does most harm where generated code is executed before a person has read it.

The syllabus asks organizations to implement four robust data privacy measures: data minimization, data anonymization and pseudonymization, secure data storage and transmission, and resources training.

Minimize first, then mask what remains

Minimization comes before masking: first ask which data the test task actually depends on. Tests of a shipping-cost calculator depend on parcel weights, dimensions and destination zones, not on recipients' names and phone numbers, so those fields stay out of the prompt. Often the model needs a description rather than data: for a library-loans system, the record layout and the loan rules can be enough to generate plausible synthetic borrowers. Data that is never sent needs no masking.

When the sensitive field is what is under test, removing it defeats the test. Testing how a system validates international phone numbers or postal codes needs values in those formats, so use invented values that follow the same structure rather than real customers' details. Anonymization removes the link to a person for good; pseudonymization swaps identifiers for stand-ins that could be re-linked through a separately held key, which is why the GDPR treats pseudonymized data as personal data for an organization that holds, or can obtain, that key.

What encryption does and does not do

Encrypting the connection to an LLM protects data in transit, and encrypting stored logs protects them at rest. Neither stops the receiving tool from retaining or reusing what it receives, and neither checks what the model generates. Access control matters as much as encryption: encrypted data is only as protected as the list of people who can read it once decrypted.

Training and policies address behaviour

Many privacy failures start with a choice a person makes: which tool to use, which data to share, whether to check the output. Training programs and usage policies act on those choices directly and promote ethical practice; technical controls limit the damage afterwards.

The syllabus lists five additional strategies to consider when implementing GenAI for testing: systematic review of the generated output, evaluation by comparison with another LLM, choice of a secure operational environment, regular security audits and vulnerability assessments, and staying updated with security best practices.

• Systematic review of the generated output. Human evaluation is essential to the quality and accuracy of GenAI-powered test tasks. It is the control that catches malicious generated code before it runs. Automated checks can point a reviewer at the outputs that most need attention, but they feed human evaluation rather than replace it.
• Evaluation by comparison with another LLM. Use several LLMs on the same task and compare their responses. Where independent models disagree, the output deserves a closer look; the comparison is an automated signal that can run on every output.
• Choice of a secure operational environment. Choose according to the level of confidentiality required (see the next section).
• Regular security audits and vulnerability assessments. Identify and address weaknesses in the GenAI systems an organization already uses.
• Staying updated with security best practices. Keep up with the latest security guidelines and technologies. Attacks on LLM tools are still evolving, so practices that were adequate when a tool was introduced may not be adequate now.

The syllabus gives three options for a secure operational environment and says the choice depends on the level of confidentiality required: a commercial secure offering from an LLM provider, operating the LLM in a secure cloud, or installing the LLM in the organization's own infrastructure.

The table is practical guidance rather than syllabus text: the syllabus itself says only that the choice depends on the confidentiality required. It is consistent with the LLMOps discussion in section 4.2.2, which describes LLM-as-a-Service as usable if the necessary assurances are given and in-house open-source LLMs as the route to greater control. Whichever option is chosen, it is one control among several, not a replacement for the others.

The syllabus states that the strategies are often complementary and that a combination of them is required to ensure data security when using GenAI. No single measure covers every risk, because the risks sit in different places: in the data sent, in the data stored, in the generated output, in people's behaviour and in the systems themselves.

So when one design has problems in more than one place, each needs its own mitigation: a measure that protects the data going in does nothing about what comes out, and a measure that checks output does nothing about where prompts are stored.

The syllabus position is that data protection regulations like the GDPR do not restrict the applications of GenAI explicitly, but they provide safeguards that may limit what can be done, particularly regarding lawfulness and limitations on the purposes of collection, processing and storage of data.

Two wrong conclusions follow from misreading this. "GDPR does not mention GenAI, so anything goes" is wrong, because pasting customer records into a GenAI tool is still processing personal data and must still be lawful and purpose-limited. "GDPR bans GenAI" is also wrong: it constrains how personal data is handled, not the use of the technology.

The syllabus highly recommends involving senior security engineers, legal counsel, the CTO or the CISO (if the organization has one) in data security decisions about GenAI. Between them these roles bring the legal, security and technology perspectives that a data security decision about GenAI needs. Other AI regulations and standards, such as the EU AI Act, are covered in the lesson on sections 3.3 and 3.4.

The hands-on objective HO-3.2.3 (H0) is to recognise data privacy and security risks in a given case study of GenAI used for testing, such as model vulnerabilities, unauthorized data access or malicious use of generated outputs, identify mitigations such as secure data handling, robust access controls and AI monitoring practices, and reflect on the ethical and practical implications. A practical way to do this is to trace a design's data flow and ask a fixed set of questions:

1. What goes into the prompts, and who controls it? Personal or confidential data points to exposure, lack of control and compliance. Content that comes from outside the team points to malicious input and request manipulation.
2. Is anything the tool receives during use kept and learned from later? If so, the training data can be manipulated, which points to data poisoning.
3. What happens to generated output before it is used? Output that is acted on without a person reading it gives malicious code generation a free path, which makes systematic review essential.
4. Where are prompts, responses and logs stored, and who can reach them? Broad access or shared storage points to breaches, unauthorized access and unintentional exposure.
5. Which environment hosts the model, and does it fit the confidentiality required?
6. How is the tool monitored in operation, and is its use acceptable to the people whose data it touches? Monitoring shows when behaviour changes after deployment; the ethical question covers what people would reasonably expect to happen to their data.

Severity grows where risks chain together: when untrusted input can shape output that is acted on with no human step in between, a single manipulation can travel all the way to impact.