Security+ SY0-801 AI Threats and Vulnerabilities

AI threats and vulnerabilities are the ways an artificial intelligence system can be attacked, misused or simply go wrong in a way that harms security, people or the organization. Security+ objective 2.6 lists thirteen of them: model manipulation, poisoning, prompt injection, data loss, bias, explainability, hallucinations, jailbreaking, evasion, privacy, ethical considerations, session hijacking and code execution. The objective asks you to summarize them, which in practice means recognizing each one from a short description of what happened.

The list mixes two kinds of item. Some are attacks: someone deliberately acts against the AI system (poisoning, model manipulation, evasion, prompt injection, jailbreaking, session hijacking and most code execution). Others are inherent weaknesses or risks of using AI that exist with no attacker at all (bias, explainability, hallucinations, data loss, privacy and ethical considerations). Asking "is there an attacker, and if so, what did they touch?" is the fastest way to tell the items apart.

A useful way to locate each attack is the AI lifecycle. A model is trained on data, the finished model is stored and deployed as a file plus its serving settings, and at inference (run time) it receives inputs and produces outputs, which an application may then act on.

The rest of this lesson takes the threats in groups of near-twins, because the exam sets the closest pairs against each other.

Poisoning, model manipulation and evasion all make a model give the wrong answer, and the difference is what the attacker changes: the training data, the model itself, or only the input.

Poisoning

Poisoning corrupts the data a model is trained or retrained on so that it learns the attacker's chosen behaviour. The data can be planted anywhere it is collected: scraped web content, a shared labeling project, or a feedback loop in which a system learns from what users tell it. For example, a product-recommendation model that retrains nightly on ratings can be pushed by a bot network leaving thousands of fake five-star ratings until it promotes the attacker's listings; a chatbot that learns from conversations can be taught offensive replies by coordinated users. A backdoor is a special case: a small number of poisoned examples teach the model to behave normally except when a rare trigger appears (an unusual sticker in an image, a specific token in a file), then to return the attacker's preferred result. The attacker uses the trigger at run time, but the attack is still poisoning, because the trigger only works since the model learned it from corrupted training data.

Model manipulation

Model manipulation alters the trained model directly, without touching the data it learned from. Examples include editing the stored weights, swapping the model file for a look-alike, or changing operating parameters in the serving configuration, such as lowering the confidence cutoff a content-moderation model needs before it removes a post so that harmful posts stay up. It requires access to where the model is stored or to the deployment pipeline. Because the model now differs from what was built, the way to detect it is to compare the deployed model and its settings with an approved, signed version.

Evasion

Evasion (an adversarial-example attack) crafts inputs at inference time so that an unchanged model misclassifies them. A spammer misspells trigger words and puts the message text inside an image so a spam classifier scores it as clean; a few small stickers on a stop sign make a vehicle's vision model read it as a speed-limit sign; a phisher pads a malicious URL with harmless-looking path segments until a URL-reputation model rates it safe. Evasion also defeats behaviour-based anomaly detectors: an intruder who exfiltrates data in small transfers during business hours, over ports the host normally uses, shapes the activity to fall inside the learned baseline, so the unchanged detector sees nothing unusual. Nothing about the model or its data is altered; only the input is.

An investigator separates these by checking integrity at each stage. Compare the training set with its trusted source, and the deployed model and configuration with the approved release. Whichever stage no longer matches is where the attack happened; if both match, the attacker can only have worked through the inputs.

Prompt injection and jailbreaking both use text to make a language model misbehave; the difference is what the text defeats. Prompt injection overrides the application's instructions so the model works for the attacker; jailbreaking defeats the model's own safety restrictions so it produces content it is built to refuse.

Prompt injection

A large language model receives its developer's instructions (often called the system prompt) and untrusted text in the same stream, and it cannot reliably tell them apart. Prompt injection exploits this by supplying text that the model treats as new instructions.

• Direct prompt injection: the user types it, for example telling a travel-booking assistant that it is now in "administrator mode" and should apply a staff discount, or asking a support bot to repeat everything it was told before the conversation started, which can reveal hidden instructions or make the bot do work it was never meant to do.
• Indirect prompt injection: the instructions are planted in content the AI processes on someone else's behalf, such as a calendar invitation a scheduling assistant reads, a PDF whose invisible text a contract-review tool ingests, a product review on a page a shopping agent visits, a comment in a code repository a coding agent works on, or a document that a retrieval-augmented (RAG) assistant pulls into its prompt when it answers. The victim never sees the instructions.

Indirect injection is most dangerous when the AI has tools and access: a coding agent that reads a planted repository comment and then pushes the project's secrets to an outside server has been turned against its own user. Tampering with a retrieval store sits close to poisoning, and some frameworks (including OWASP's LLM Top 10) group it with data poisoning and vector-store weaknesses. The useful distinction is the mechanism: when planted instructions are read into the prompt and followed at run time, with the model's training untouched, that is prompt injection; corrupting what the model learns is poisoning.

Jailbreaking

Jailbreaking is a user manipulating a model so it bypasses the safety restrictions built into it. Common techniques are role-play personas, fictional or hypothetical framing, and fragmentation: splitting a refused request into harmless-looking pieces that the model assembles. The jailbreaker works in their own session and targets the model's refusals, not an application's task.

Neither one changes the model's weights, so neither is model manipulation, and neither touches training data, so neither is poisoning. Mitigations include treating all retrieved and user content as untrusted, separating and limiting what the AI's tools can do, requiring human approval for sensitive actions, and filtering inputs and outputs (guardrails and other controls are covered in the security operations objectives).

Session hijacking and code execution are familiar attacks that take on new forms when AI assistants and agents hold sessions, connect to tools and run code.

Session hijacking

Session hijacking against an AI system means taking over another user's authenticated assistant or agent session so the attacker acts as that user, with their conversation history and every tool and data source connected to the session. Routes in include a session token captured by an adversary-in-the-middle phishing proxy, an access token left in a URL or a shared notebook and reused, and predictable or reused session identifiers: if the server that connects agents to their tools hands out guessable IDs, an attacker can claim another client's session and feed it crafted responses that the victim's assistant then acts on. A 2025 vulnerability in an agent tool-protocol library (CVE-2025-6515) worked this way, because its session IDs could be reused and predicted. The attacker's requests may look ordinary; they succeed because of whose session they are in. Defences are the usual session controls applied to AI: unpredictable session IDs, short-lived tokens bound to the client, never logging or sharing secrets, and re-authentication for sensitive tool use.

The data exposed afterwards is a consequence; the attack itself is the session takeover. And if the planted responses steer the agent, the root vulnerability is still the session that let them in.

Code execution

Code execution is an AI system running code it should not. Two patterns dominate:

• Unsafe model files. Some model formats are serialized with libraries (such as Python's pickle) that can run arbitrary code when the file is loaded. A shared or downloaded model can therefore install a cryptominer or create a backdoor account the moment it is loaded, and still make reasonable predictions afterwards, so testing its accuracy reveals nothing. Treat model files like executables: use trusted sources, verify signatures and hashes, scan them, and prefer formats that store only data (such as safetensors).
• Executing model output. An application that feeds model output into eval(), a database query, a shell or an interpreter, or runs model-written code in its own server process, lets anyone who can shape the output run commands. A code sandbox only helps if it is isolated: running as root, sharing the host's file system or network, or exposing a host control interface (such as a container runtime's socket) lets model-written code escape and act on the host.

Writing malicious code is not the same as executing it: a model that is talked into producing harmful code has been jailbroken, and code execution only occurs when a system runs code. The design rule is to treat model output as untrusted input, validate it, and run any generated code with least privilege in a genuinely isolated environment.

A hallucination is output that a generative AI model states confidently but that is false or has no basis in its sources. It happens without any attacker, because language models generate plausible text rather than looking facts up. Typical forms are a CVE number or patch version that does not exist, a court case or standard cited in a policy draft that was never published, a claim in a document summary that the source does not contain, and a confident statement about an incident (an event, a host, a timestamp) that the underlying records do not show.

Hallucinations become a security threat in two ways. First, people act on them: a responder who closes an incident because the AI said a patch was applied, or an engineer who pastes an invented configuration setting, makes decisions on false information. Second, attackers exploit predictable ones. When coding assistants repeatedly suggest a dependency name that does not exist, an attacker can register that name in a public registry with malicious code, so developers who follow the suggestion install malware (sometimes called "slopsquatting"). The attacker never touched the models; they waited for a fabrication and claimed it.

The control is verification: check AI answers against authoritative sources, confirm that suggested packages exist and are trusted before installing them, and take evidence from the original system rather than from the AI's summary of it. A hallucination is a single fabricated output; it is not explained by poisoning, which shifts what a model learns over time, or by evasion, which makes a model miss crafted input.

Bias and explainability are both about trusting AI decisions, but they answer different questions: bias is whether the outcomes are fair and accurate across groups; explainability is whether anyone can trace how a particular outcome was reached.

Bias

Bias is a systematically skewed or unfair result, most often caused by training data that does not represent the population the model serves, though it can also enter through how data is labeled, which features are chosen and how the model is designed. No attacker is needed. A skin-condition model trained mostly on images of light skin misses conditions on darker skin; a speech-to-text tool used for call-center quality scoring makes more errors for non-native speakers and so scores them lower; a fraud model trained mostly on transactions from customers the bank already watched closely flags that group more often. Data can be genuine and intact and still biased: a long record of real promotion decisions that favoured one group teach a promotion model to repeat that pattern, even though every record is accurate.

The test that separates bias from poisoning is intent and integrity: poisoning is deliberate tampering by an attacker, while bias comes from what honest data and design represent. If the training set is exactly what was collected and no one altered it, a skewed outcome is bias.

Explainability

Explainability is the ability to trace and justify how a model produced a specific output; the security concern is its absence. Complex models can be accurate overall while no one can say why they made this decision. That blocks regulated or contested decisions (a customer appealing an insurance claim the model rejected), audits and investigations: if a security tool's AI auto-closed an alert that later proved to be a breach and no one can say why, the team has no way to trace the reasoning behind that single decision. A model's headline performance score says nothing about one outcome; per-decision explanations, such as feature-attribution reports, do. Where those explanations exist and show what drives the decisions, the decision is explainable, and any unfairness they reveal is a separate, bias problem.

Data loss and privacy both involve information escaping through AI, and the line between them is whose information it is. Data loss is sensitive organizational data leaving the organization's control; privacy concerns individuals' personal data and their rights over it.

Data loss

AI creates new exits for confidential data:

• Public AI tools: staff paste proprietary source code, draft contracts or financial forecasts into a consumer AI service whose terms let the provider keep and learn from what users submit.
• Over-permissioned assistants: an enterprise assistant that indexes every SharePoint site and HR folder answers any employee's question with details from salary spreadsheets or board papers outside that person's permissions. An AI assistant should see no more than the user it is answering.
• AI features that share: a note-taking bot that posts call summaries to a channel with outside members; a browser AI extension that sends the contents of every page it reads to a third-party service; a conversation made public through a sharing feature.

Contract terms about how the vendor uses prompts cover only one path; sharing, integrations and extensions remain separate routes for data to leave. Controls include approved enterprise AI services, data loss prevention, sharing restrictions, and permission-aware retrieval.

Privacy

AI raises specific privacy problems for people whose data a model touches:

• Data-subject rights: once a person's emails or chats have been used to train a model, deleting the original records does not remove what the model learned; personal data built into a trained model is hard to remove, so honouring erasure requests is difficult. Decide before training on personal data.
• Memorization: a model trained on internal records can reproduce real people's phone numbers, account details or health information word for word when prompted the right way. The output is real data, not a hallucination.
• Inference attacks: membership inference uses a model's outputs and confidence scores to determine whether a particular person's record was in the training set. For a model trained only on, say, participants in an addiction-treatment program, that alone reveals sensitive facts about the person. Nothing is exfiltrated as a document; the personal information leaks through what the model's answers reveal.

Ethical considerations ask whether a use of AI is right and fair to the people it affects, which is a separate question from whether it is legal, accurate or unbiased. The main themes are transparency (do people know AI is involved?), consent, accountability (who answers for the decision?) and deception.

Ethics is the concern left standing when the other items do not explain the problem. A company might deploy an AI voice agent that phones customers about overdue bills without saying it is not a person, in a place where no law requires that disclosure. A school might run AI analysis of students' writing to predict who is likely to drop out and quietly change how teachers treat them, with accurate, unbiased predictions but no notice to students or parents. A firm might use a departed employee's recorded voice to train a synthetic voice for its phone menus because the employment contract technically allowed it. Each use may be lawful and technically sound, yet people are deceived, uninformed or denied a say. (Some deceptive uses are also illegal, such as presenting AI-written reviews as genuine customer testimonials in the US; then compliance and ethics both apply.)

Legal is not the same as ethical: a lawful, unbiased and explainable AI use can still be unethical if people are deceived about it or never had a chance to know it was used.

Most objective 2.6 scenarios can be classified with a short sequence of questions, asked in order.

1. Is there an attacker? If not, choose among bias (unfair across groups), explainability (cannot say why), hallucinations (confident but false), data loss (company data escapes), privacy (personal data or rights) and ethical considerations (legal but not right).
2. Did the attacker change what the model learns? Training data, labels or feedback: poisoning, including trigger backdoors.
3. Did they change the model itself? Weights, model file or thresholds, with clean data: model manipulation.
4. Did they only shape inputs to an unchanged model? Evasion, whether against a classifier or a behaviour-based detector.
5. Did text make a language model follow new instructions? Overriding the application's task, typed or hidden in processed content: prompt injection. Defeating the model's own safety rules: jailbreaking.
6. Did they take over someone else's session or token? Session hijacking.
7. Did a system run code it should not? Loading an unsafe model file or executing model output: code execution.

Watch for the consequence trap: an attack often results in another item (a hijacked session exposes data; a prompt injection leaks information). The answer is the mechanism the scenario describes, not the downstream harm, unless the question asks about the harm. Evidence matters as much as the symptom: a training set or model shown to be unchanged rules tampering at that stage out, and a fact established by audit (lawful, fair, explainable) shifts the question to whatever is still unexplained.

Reducing AI risk applies existing security principles to each stage of the AI lifecycle rather than relying on one product.

• Data: trusted sources and provenance for training data, validation of labels and feedback, representative datasets checked for bias, and minimizing personal data before training.
• Model: sign and hash model files, verify them before deployment and loading, prefer data-only model formats, and control who can change models and serving configuration.
• Inputs and outputs: treat prompts, retrieved content and model output as untrusted; never pass output straight to a shell; verify important claims against primary sources.
• Access: give assistants and agents least privilege and permission-aware retrieval, require approval for sensitive actions, and protect sessions with unpredictable IDs and short-lived tokens.
• Governance: acceptable-use policies for public AI tools, approved enterprise services, transparency to the people affected, and per-decision explanations where decisions are regulated.

Specific AI security controls and AI-assisted security operations are covered in later objectives; objective 2.6 focuses on recognizing the threats themselves.