CCSP AI and ML Data Set and Model Privacy and Security

Protecting AI and machine learning data means applying the cloud data security programme to everything a model is built from and everything built from the data: training, validation and test data sets, feature stores, embeddings and vector indexes, prompts and outputs, and the trained model itself. CCSP sub-domain 2.9 has two official bullets, and they pull in two directions:

• Data set and model privacy: personal data must be lawful to use for training, minimized before it gets there, and not leak back out through the model, its outputs, its derived artefacts or the AI services it is sent to.
• Data set and model security, with validation and verification as the named examples: the data and the model must be correct, fit for purpose, unaltered and traceable from the data that trained it to the version running in production.

The key idea is that a model is not a neutral piece of software sitting beside the data. It is derived from the data, so it can carry the data's content, its legal obligations and its errors. The table maps the main artefacts to the risk each carries.

Sub-domain 1.6, Comprehend Artificial Intelligence (AI)/Machine Learning (ML), is about AI used for security: cloud threat detection and analysis, data source validation and verification, SOAR, ethical concerns and regulatory requirements. The 1.6 lesson also covers attacks on models such as poisoning, evasion and membership inference. This lesson takes the data protection angle: classification, lawful use, minimization, privacy-preserving training, erasure, provider terms, retrieval permissions, and the validation, verification and integrity controls of the ML pipeline. Pseudonymization and anonymization techniques themselves are taught with obfuscation in sub-domain 2.3.

Anything derived from sensitive data inherits at least the classification of its source, and that includes training sets, feature stores, embeddings and fine-tuned model weights. A model fine-tuned on restricted merger-negotiation emails is restricted, even though nobody can open the weights file and read an email in it.

The reason is that derived artefacts can give the content back:

• Model weights can memorize training records. Through ordinary prompting or deliberate extraction attacks, a model can reproduce passages, names or numbers it was trained on.
• Text embeddings are arrays of numbers, but research on embedding inversion shows the original text can be largely reconstructed from them, and nearest-neighbour searches reveal which documents discuss what. An embedding store built from patient complaint letters is, for protection purposes, a copy of the complaint letters.
• Feature stores and training extracts are often literal copies of production data, moved to an environment with looser access.

The rule to apply is that classification follows the content an artefact can reveal, not the form it is stored in or the team that produced it. A change of representation (text to vectors, rows to weights) is not de-identification, and the data owner, not the artefact's builder, sets the level. Treatments that leave the content intact do not lower it either: removing file metadata such as authors and file names before embedding leaves the body text, and the body text is what the embedding encodes.

Inheritance has practical consequences. Access to the vector database or model registry is limited to people cleared for the source data, on the same least-privilege basis; the artefact needs the same encryption, logging, residency and retention handling; and it appears in the data inventory so discovery and classification (sub-domains 2.4 and 2.5) cover it. Protection has to keep the artefact usable, so the controls are on who reaches it and how it is stored, not on scrambling the vectors themselves.

A model can be classified lower than its training data only when the data itself has been made lower risk first, for example by removing identifiers from the text before training or by training with differential privacy, and that decision is documented by the data owner.

Personal data can be used to train a model only when that use is lawful for the purpose, and then only the data the purpose needs, kept no longer than needed and kept accurate. These privacy principles come before any technical design.

Storage limitation and accuracy matter for ML too (retained training snapshots need a justified retention period; a model trained on wrong records about people makes wrong decisions about them), but neither answers whether a new use is allowed. That is purpose limitation's question.

Purpose limitation comes first

Purpose limitation means personal data collected for a stated purpose is not reused for an unrelated purpose without fresh justification. Training a new model is a purpose in its own right. Badge-access logs collected for building security, for example, cannot simply be fed into a model that scores employee productivity. The organisation must first assess whether the new purpose is compatible with the original one and, if it is not, establish a new lawful basis (such as consent) and update the privacy notice so data subjects are informed. A privacy impact assessment belongs before the processing starts, because its findings may change or stop the processing.

Technical and access controls do not change this answer. They decide how permitted processing is protected; they do not make unpermitted processing lawful. Pseudonymized data is still personal data: replacing direct identifiers with tokens leaves the records linkable to people for the organisation that holds or can reach the token mapping, so purpose limitation still applies. Only data that has been genuinely anonymized falls outside it.

Then minimize, at the source

Data minimization limits the training data to what the model's stated purpose needs. A model forecasting call-centre staffing needs call volumes, durations and timestamps, not callers' names or account balances. The strongest form is not to extract unneeded fields at all: protecting a field the purpose does not need, however well, still leaves the team holding personal data it has no reason to have.

Minimization has a trap in unstructured data. People write names, policy numbers and addresses into claim descriptions, emails and clinical notes, so removing the identifier columns leaves the identifiers in the text. For free text, the fix is content-level detection and redaction of identifiers before the text enters the training pipeline. Once the identifiers are in the training data, the model can learn them, and controls applied later work around a problem that is already built in.

Privacy-preserving training techniques reduce how much training data has to move or how much a model reveals about any one record. The two this topic covers are federated learning and differential privacy, and they are often combined.

Federated learning

In federated learning, each participant (an organisation, a branch, a user's phone) trains on its own data locally and sends only model updates to a coordinator, which aggregates them into a shared model. Raw records never leave each participant's environment. It is the approach for parties that want a shared model but cannot pool their data, for example several insurers building a claims-fraud model. It is defined by where the data stays; most other named learning methods, such as transfer learning (reusing a model trained for another task), describe how a model learns and say nothing about data location.

Federated learning does not remove privacy risk on its own. Model updates are computed from the records, and research on gradient inversion shows updates can be analysed to infer or reconstruct training data. The coordinator, often hosted by a cloud provider, is therefore still a party that could learn about each participant's customers. Two controls close the gap:

• Secure aggregation combines updates cryptographically so the coordinator sees only the aggregate, never an individual participant's update.
• Differential privacy adds calibrated noise so no update, or final model, reveals much about any single record.

Protection on the network path does not help against this risk, because the coordinator is the intended recipient and sees each update once it is decrypted.

Differential privacy and epsilon

Differential privacy is a mathematical guarantee that the result of a computation (here, the trained model) changes only a bounded amount whether or not any one individual's record is included. The bound is the privacy budget, epsilon.

Choosing epsilon is a privacy-utility trade-off made jointly by the privacy officer and the data owners, not a purely technical setting. Differential privacy also defends against membership inference (see the 1.6 lesson), but it does not cancel legal obligations such as erasure requests.

Trained models and synthetic data sets can leak personal data because a generative model trained on real records can memorize them and reproduce them in its output. Removing a file from view does not remove what was learned.

Memorization in trained models

Memorization is diagnosed by where the data could have come from. Suppose an internal coding assistant, fine-tuned on the company's repositories and given no access to live systems or search, completes a snippet with a real database password that once sat in a configuration file. The only source it had is its training data: the model memorized it, which also shows the secret was never removed before training. Keep the neighbouring terms apart:

• Memorization: real training content reproduced in outputs, a privacy failure of the training data.
• Poisoning: training data deliberately manipulated to change behaviour.
• Evasion: inputs crafted at inference time to cause a wrong result.
• Unrepresentative data: training data unlike production, which hurts accuracy but does not plant real records in outputs.

The fix works at two layers:

• Root cause: retrain on de-identified data. Retraining on the same data reproduces the same model behaviour.
• Interim exposure: if the model must run before retraining finishes, the interim control has to act where the leak happens, on the outputs: scan responses and block personal data before they reach users. The problem lives in the weights, so controls on other layers leave what the model says unchanged, and an instruction to the model is guidance it may not follow, not an enforcement control.

Synthetic data is not automatically anonymous

Synthetic data generated by a model trained on real records looks artificial, but the generator can reproduce real individuals, and people with rare combinations of attributes (outliers) are the most exposed. Not copying rows directly is not a privacy argument. Before release, the synthetic set needs privacy testing that measures disclosure directly, by attacking the data the way an adversary would: looking for synthetic records that sit implausibly close to a real person, or attempting to re-identify individuals from them.

Keep two questions apart. Is the data useful? is answered by utility measures, which compare the synthetic set's behaviour with the real one. Does the data disclose anyone? is answered only by a disclosure test.

High utility can coincide with leakage, so a good fidelity score proves nothing about privacy. When testing finds near-identical rows, the leak is a property of the generator, so the whole set is potentially personal data: the fix is to regenerate with controls such as differential privacy and test again.

An erasure request is not fully handled when the source records are deleted: a model trained on those records may still hold information learned from them. Suppose a streaming service deletes a subscriber's viewing history, including every stored copy, after an erasure request. A recommendation model trained on that history months earlier is a separate artefact, and deleting stored data does nothing to it.

The privacy team therefore has to assess whether each affected model retains information about the person and decide on a remedy. The usual documented remedy is retraining from the cleaned data; techniques for "machine unlearning" without full retraining are still emerging and should not be assumed. Training with differential privacy reduces what a model reveals about any one person, but it does not make the erasure obligation disappear.

Meeting legal deadlines depends on lineage: records linking each data set version to every model version trained from it. When source data changes constantly and models retrain on a schedule, lineage is the only way to answer "which models were trained on this person's records?" quickly, and it scopes retraining to the models actually affected. Erasure tools that act on storage, such as crypto-shredding with per-subject keys, remove stored copies efficiently but cannot reach what a model learned from decrypted data. Lineage also needs the training-set versions it points to, which is why snapshots are retained (under a justified retention period) rather than discarded after each run.

Lineage and erasure workflows connect to retention and deletion policy (sub-domain 2.7) and to the auditability of data events (sub-domain 2.8).

Data sent to a provider-hosted AI service is protected mainly by contract and data processing terms, and data reached by a retrieval assistant is protected by enforcing the source documents' permissions at retrieval time. Both are frequent cloud scenarios.

Provider-hosted AI service terms

Before staff submit confidential content, such as an engineering firm's unreleased design specifications, to a generative AI service, the first term to confirm is whether the provider uses customer prompts and outputs to train or improve its models. If it does, the content leaves the customer's control. Security assurances describe how the provider protects content; the data-use terms decide what it may do with it, so strong assurances cannot make up for a training-use clause.

A no-training clause answers only one question. The agreement must also be checked for:

• Retention: providers may keep prompts and outputs for a period, for example for abuse monitoring, even when they never train on them.
• Location of retained data: where retained prompts are stored is a separate fact from where requests are processed, and it has to be stated in the service's terms if residency rules apply. ISO/IEC 27018 is a code of practice for protecting PII in public cloud processing; it does not fix where data is stored.
• Subprocessors, deletion on request and breach notification, as for any processor handling personal data.

Permission-aware retrieval

In retrieval-augmented generation (RAG), an assistant searches an index of documents and passes the relevant passages to the model to answer. If retrieval runs under one identity with broad read access, a sales representative asking about headcount can be shown passages from a draft restructuring plan that only HR can open: the repository's permissions held, but the retrieval step bypassed them. That is an authorization failure, so only an authorization control at retrieval fixes it; controls that protect storage or limit query volume do not.

The control is to filter candidate passages by the asking user's current entitlements before any content reaches the model. What that requires:

Coarse partitions or periodic rebuilds trade correctness for convenience, and reviewing answers after generation cannot work, because a reviewer cannot know each user's entitlements.

Verification confirms a data set or model was built as specified; validation confirms it is fit for its intended use, measured on data that represents the conditions it will face. Verification asks "did we build it right?" and validation asks "did we build the right thing?". The official outline names both as examples of data set and model security.

A model can pass every verification check and still perform badly in use, which only validation reveals. Accreditation and attestation are governance outcomes that may rely on both; they are not tests themselves. Calibration (aligning a model's predicted confidence with how often it is actually correct) and benchmarking against a peer model are narrower activities.

Validation needs independent evaluation data

A validation result means something only if the evaluation data is independent of the training data. When the two overlap, the model is partly being tested on records it has seen, so the score measures memorization and overstates real accuracy. This data leakage appears in several forms:

• Near-duplicates: splitting a collection into training and test sets assumes its records are independent. In a support-ticket archive where customers resubmit the same complaint weeks apart, copies of training tickets land in the test set whether the split is random or by date, because a copy can be created at any time. Near-duplicates must be found and removed across the split before the score is trusted.
• Test-set reuse during tuning: choosing settings round after round by the test-set score lets information about the test set flow into the model, so the final score is optimistic. Tune against a separate tuning set (ML practitioners call it the "validation set", a different use of the word from the assurance activity above) and use the final test set once.

Adding more evaluation data with the same flaw only gives a more precise estimate of the wrong number. Validation of data sources feeding security-operations models is covered in sub-domain 1.6.

Training data is protected in the pipeline by two controls: automated validation gates that check content before it enters training, and immutable, versioned snapshots that keep the exact data each model was trained on.

Validation gates check content, not origin

A data validation gate checks each incoming feed's schema, value ranges and distribution against expectations and history before the data is used. It runs per source, before training, so a bad feed is caught at the source rather than after it has degraded the model. Consider a fleet of building sensors where one site's firmware update switches temperatures from Celsius to Fahrenheit. The data is genuine: it comes from the real sensors, unaltered, over an authenticated channel. Origin and integrity controls prove who sent data and that it was not changed in transit, so they pass it; only a check of the content catches it. Review of the finished model comes after the damage and depends on someone noticing.

When a gate fires, the aim is to contain the suspect input without stopping good work or altering data whose cause is unknown: quarantine the flagged data for review, continue with the data that passed if the schedule requires a release, and fix the cause before the quarantined data is used. A gate whose findings can be overridden by including the data anyway no longer protects anything.

Versioned snapshots make models reproducible

Verification and audit both need the exact data that trained a model. If the training table is edited in place, nobody can reproduce a run or check what a model learned from, months or years later. The control is a write-once (immutable) data set snapshot for each training run, recorded in the model's metadata by snapshot ID and content hash, and retained for as long as the model is in use or regulation requires. Each property does a job:

Controls built for other purposes, such as backups (recovery), access logs (which record events, not contents) and encryption (confidentiality), do not provide reproducibility on their own. Snapshots also feed the lineage that erasure handling depends on.

Model artefacts are protected by digital signatures verified at deployment, by treating third-party model files as untrusted code, and by separation of duties so that no single person can put a model into production.

Sign approved models, and protect the key

To make deployment refuse any model altered after approval, even by someone with write access to the registry, sign the model at approval and verify the signature at deployment. A plain hash stored beside the file is only as trustworthy as the storage holding it: a registry writer can replace both. Controls that protect or recover the stored file do not stop an altered file being deployed.

A signature is only as strong as the custody of its private key. If everyone who could tamper with a model can also use the signing key, the signature proves nothing about approval. Hold the key in a key management service or HSM that only the approval step can invoke, so a valid signature means the approval step really ran.

Treat downloaded model files as code

Some model serialization formats (such as Python's pickle, used by several frameworks) rebuild objects by running code, so loading the file runs whatever the file contains. That gives this attack a distinctive timing: it acts at load, before the model is ever queried, for example as unexpected processes or outbound connections from the environment that opened the file. Attacks that work through a model's learned behaviour or its inputs only show up once the model is used.

Provenance signals, such as a publisher's reputation or a matching checksum, show who published a file and that it is unchanged; neither says what the file does when it is loaded. Before using a third-party model with internal data, load it in an isolated environment and convert it to a weights-only format that stores tensors and metadata without executable code, then use only the converted file. Some frameworks now default to a restricted, weights-only loader for such checkpoints, which helps, but the format itself still permits code execution, so the file is treated as untrusted code regardless.

Separate duties across data, training and promotion

When the same people can change training data, start training runs and promote models without review, one person can put a tampered model into production. Separation of duties splits those steps between roles, with independent approval at promotion, so tampering needs collusion. It is easy to confuse with neighbouring principles:

Small teams can meet a preventive requirement without new staff by having the pipeline enforce it: promotion is blocked until someone other than the change author approves. The test of any proposed control is whether it makes two different people necessary at the moment of promotion; controls that review activity afterwards are detective, and controls that only narrow when or how much access someone holds still let one person do both steps.