CySA+ CS0-004 AI in Security Operations: Risks, Governance and SOC Use Cases

Objective 1.6 asks you to summarize three things about AI in a security operations center (SOC): the risks AI introduces (hallucinations, data exposure, model poisoning and malicious prompts), the governance that controls its use (legal or regulatory compliance and AI usage policies), and the use cases where it helps analysts (comparing artifacts, analyzing log files, document creation, incident investigation, event correlation, and automation and orchestration). It is new in CS0-004 and it is a "summarize" objective: you are expected to recognise each risk, match each control to the risk it addresses, and decide where AI output can be trusted and where a person or a deterministic tool must check it.

Two kinds of AI appear in the SOC, and the difference explains most of the objective:

A recurring theme follows from that table: AI speeds analysts up, but it does not replace evidence. Hashes, diffs, raw logs and authoritative databases stay the source of truth; AI explains, drafts and prioritises on top of them. The SOAR mechanics themselves (playbooks, APIs, workflow design) belong to objective 1.5; this lesson covers what changes when AI is part of that process.

A hallucination is AI output that is fluent and plausible but not supported by any real data: an invented CVE ID, a patch number that the vendor never published, a log event that never happened, or a lateral-movement path that appears in no record. LLMs generate the most likely next text, so a well-formed identifier is easy for them to produce whether or not it exists. Hallucinations are a property of the model's output, not an attack: nobody had to tamper with anything.

Tell it apart from its near-twins:

• Hallucination invents something that does not exist.
• A training cutoff makes the model omit real things that are newer than its training data; it does not invent them.
• A coverage gap is a true answer about incomplete data: an assistant that searches only endpoint telemetry can correctly report "no sign of credential dumping" on servers that never had the EDR agent installed. That is not a hallucination, and it does not prove the event did not happen; the question stays open until the missing sources are reviewed.
• Poisoning is a faithful result of corrupted training data, and prompt injection is the model following malicious instructions in its input.

Controls that actually reduce hallucinations

• Grounding with retrieval (retrieval-augmented generation, RAG): fetch the authoritative data, such as the current asset inventory, policy export or the relevant records, and place it in the prompt so the model answers from it. For private, fast-changing facts this is better than fine-tuning, which bakes facts into the model, needs retraining whenever they change and is stale between runs.
• Citations to the evidence: require each finding to point at the record behind it, so an unsupported claim stands out.
• Human verification of key findings against the original data, and checking every identifier (CVE, KB, hash, IP) in its authoritative source such as NVD or the vendor advisory.

The common thread: hallucinations are reduced by giving the model the facts and checking what it says. Generation settings such as temperature change how varied the output is, not whether it is true, and a model's own statement of confidence is not evidence. A model that has never seen your private data cannot answer reliably about it, however capable it is.

Data exposure is sensitive information leaving the organisation's control through an AI tool. In a SOC that data is unusually sensitive: usernames, IP addresses, customer email addresses, internal hostnames, protected health information, and secrets such as API keys found in scripts. It leaks three ways:

1. Through the prompt to a provider whose terms allow prompts to be retained or used to train the model, typically a consumer chatbot or an unapproved browser extension.
2. Through provider retention and logging, even when training is off.
3. Through over-broad retrieval inside the company: an internal assistant that searches a document repository with its own broad access can return excerpts from legal-hold or merger documents to an employee who could never open those files directly.

The principle is to control exposure at the source, before the data reaches the model. Three consequences follow. Anything that acts on the model or its output after the prompt was sent cannot take a disclosure back. User-side housekeeping in a consumer tool does not change the provider's terms: whatever those terms allow it to retain or use has already been allowed. And an instruction to the model is not an access control; permissions have to be enforced by the retrieval layer itself. The best answer usually removes the exposure while keeping the business able to use AI.

Model poisoning is corrupting the data or labels a model learns from so that it learns the wrong behaviour. It targets the training stage, and it matters most for SOC models that keep learning from data an attacker can influence.

• Baseline poisoning ("boiling the frog"): an anomaly detector that retrains on recent traffic learns that slowly increasing beaconing is normal, and stops flagging it.
• Label poisoning through feedback: a phishing classifier retrained on users' Report phishing / Not phishing clicks can be taught to ignore a campaign when compromised mailboxes submit false labels.
• Backdoor poisoning: harmless samples carrying a trigger (an unusual byte string, a phrase) are added to a shared training repository labelled benign. The model then classifies anything with the trigger as benign, yet still scores highly on a clean held-out test set, because the test data contains no trigger. High accuracy does not rule out a backdoor.

Evasion is the near-twin: the attacker crafts an input at prediction time that the trained model misclassifies, without touching the training data. NIST's adversarial machine learning taxonomy (AI 100-2) separates the two by lifecycle stage: poisoning at training, evasion at deployment. If the trigger arrived through the training source, it is poisoning.

Defences

• Validate training data and labels before use: analyst review or sampling of user-submitted labels, agreement from trusted reporters, provenance checks on shared repositories.
• Control who and what can write to training sources, and keep versioned, known-good datasets and models to roll back to.
• Watch for drift: a baseline that steadily absorbs one host's growing traffic is a signal in itself.

Poisoning is an integrity attack on the learning pipeline, so the control has to check what goes into training while letting the model keep learning. Note that retraining more often does not help on its own: it just absorbs poisoned data sooner.

A malicious prompt is input crafted to make an AI system ignore its instructions or do something its operator did not intend. The OWASP Top 10 for LLM Applications (2025) lists prompt injection first and splits it in two:

Indirect injection is the bigger SOC risk because SOC data is attacker-influenced by nature: alerts quote attacker emails, URLs and headers. When an AI agent can call SOAR to disable accounts or isolate hosts, injected text can become an action. No filter or model is injection-proof, so the reliable controls bound the consequences: scope the agent's API credentials to only the actions its tasks need, and require an analyst to approve containment. Anything that only asks the model to behave reduces the likelihood at best, and a control that records what happened detects an attack without preventing it.

Sending SOC data to an AI service is a disclosure to a third party, so the same laws that govern any outsourcing of that data apply, and they must be met before the data is shared. Security logs are full of regulated data: usernames and IP addresses are personal data, and a hospital's incident data can contain protected health information.

Two traps recur. Pseudonymized data is still personal data under GDPR for the organisation that holds the means to re-identify it, so replacing identifiers with tokens or hashes keeps the data in scope. And good vendor diligence (security attestations, favourable contract terms) is valuable but never replaces an agreement or transfer basis the law requires. Industry rules such as PCI DSS and sector or national AI regulations can add further obligations; the compliance or privacy function (for example the DPO) decides what applies.

An AI usage policy (often called an AI acceptable use policy) is the governance document that states which AI tools are approved, what data may be entered into them, and when AI output must be reviewed by a person before it is used. It is the answer to shadow AI: analysts using unapproved chatbots or browser extensions and each deciding for themselves what to paste in.

Each policy clause controls a specific risk:

A policy says what is allowed and required. Technical settings and task-level detail belong in standards, configurations and procedures, which change far more often than policy. The AI usage policy sits in a family of governance documents: it relies on the organisation's classification scheme for its data labels, while assessments of individual AI systems and procedures for handling incidents are separate documents with narrower jobs.

AI helps with comparing artifacts (scripts, configurations, baselines, binaries' behaviour) by explaining what a difference means, and with analyzing log files by turning plain-language questions into queries and summarizing results. In both, it speeds up interpretation; it is not a reliable way to prove that something is absent.

Comparing artifacts

• Deterministic evidence wins. If two files have different SHA-256 hashes, they differ, whatever an AI says. When an assistant says a modified binary, scheduled task or configuration file is "equivalent" to the approved version but the hashes differ, compare the files deterministically, review every change yourself, and treat the modified copy as potential evidence.
• Context windows truncate silently. An export larger than the model's context window may be accepted without an error, and anything past the limit is never read, so "no deviations" only covers what fit. Chunk large artifacts or compare them with deterministic tools.
• The scalable pattern: run an exact diff (or hash comparison) against the golden baseline to detect every deviation, then send only the differences to the LLM to explain in plain language. That gives complete detection plus readable explanations at a fraction of the token cost.

Analyzing log files

• Natural-language querying: a SIEM assistant converts "new local administrator accounts created on file servers this month" into the SIEM's query language, which helps analysts who have not learned the syntax.
• Validate the generated query before trusting its result, especially an empty one: check field names (is the account in the field the query used?), filters, time zone and business-hours logic, the index and the time range. Only checking the query itself validates the result; the model's opinion of its own query does not.
• Scale: an LLM cannot read terabytes of logs a day within a context window or a sensible token budget, and sampling misses rare events. Use ML anomaly detection over all the data to find outliers, and use the LLM to explain those outliers to analysts.

AI is useful for document creation (drafting playbooks, incident reports, shift handovers and executive summaries) and for incident investigation (summarizing evidence, building timelines, extracting indicators), as long as people validate its output and the original evidence remains the record.

Document creation

• Rewriting technical material for a different audience, such as a customer notification or a regulator-facing summary, is document creation: the AI rewrites and condenses for a reader.
• AI drafts are generic. A playbook draft may name tools the company does not have or skip internal escalation contacts, so a knowledgeable person checks every step against the real environment before it is published. Only someone who knows the environment can test that.
• For consistent, complete drafts without training a custom model, put a fixed template and the source data (the actual records the document must cover) in the prompt. Missing items are a missing-data problem, and no setting can make the model report facts it was never given.

Incident investigation

• AI conclusions are not evidence. A copilot's timeline speeds analysis, but for possible litigation you preserve the underlying logs and artifacts behind each entry, with chain of custody. Anything that captures only the AI's output records its work, not the events.
• Verify extracted indicators before they drive automated blocking. An AI may misread a report, invent an indicator, or extract an IP or domain that belongs to shared infrastructure such as a CDN or cloud provider. Check each against the source evidence and allowlists first.
• Check coverage before concluding: an answer about the data a copilot can see says nothing about data it cannot.

In event correlation, AI and ML group related alerts into a single incident so analysts open fewer items; in automation and orchestration, AI triages alerts, selects the matching response playbook and starts it through the SOAR platform's APIs. Both cut workload, and both need guardrails so a model mistake does not hide an attack or cause an outage.

Event correlation

• Reduce what analysts must open, not what they can see. Merge alerts into incidents while keeping every source alert linked and searchable, so a wrongly merged or low-scored alert can still be found. Any setting that removes alerts from view trades visibility for workload.
• Correlate on the right entities. Grouping on one shared attribute over-merges: if remote staff all connect through one VPN concentrator or NAT gateway, grouping by its address merges unrelated users' alerts. The fix is richer entities (user identity, hostname, device) rather than less data.
• Rigid rules such as "same IP only" miss attacks that move across hosts and accounts; that is the gap ML-driven grouping is meant to close.

Automation and orchestration

• The line between the two use cases: correlation decides which alerts belong together; automation and orchestration decides and does something about them, such as selecting and running a response through SOAR.
• Human-in-the-loop by impact: automate where a mistake is cheap and require approval where it is costly. For example, an agent might reset a standard user's session automatically after a confirmed malicious sign-in, but need an analyst's approval before disabling a service account that a payroll system depends on. The deciding factor is the cost of a wrong action on that asset, not how sure the model says it is.
• Give AI agents least-privilege credentials, because any text they read may carry injected instructions (see malicious prompts above).