CCSP: Cloud Data Discovery for Structured, Unstructured and Semi-Structured Data

Data discovery is the process of finding where an organisation's data is stored and learning what it contains, so that it can then be classified and protected. It comes first in the data security process because you cannot classify what you have not found, and you cannot choose controls for data whose content you do not know.

1. Discover: locate the data stores and inspect what each one holds.
2. Classify: assign a sensitivity level under the classification policy (sub-domain 2.5).
3. Map and protect: map each data type to the controls it needs, such as encryption, DLP, masking or rights management (sub-domains 2.3 and 2.6).

The order matters most when a requirement arrives before the facts. Suppose a new privacy regulation starts to cover customers' biometric templates and the organisation has dozens of cloud accounts and workspaces where such data might sit. Any control chosen before the data is located is aimed at a guess: it is either applied too widely, at needless cost, or too narrowly, leaving regulated data outside it. The first step is to find where the regulated data actually resides; the controls are then scoped to what discovery found.

In the cloud, discovery has three recurring difficulties: knowing where the data is (self-service provisioning, many copies, unsanctioned services), getting access to it (the service model and the contract decide what the customer may scan), and keeping the results current as stores are created and changed every day. The rest of this lesson takes each of them in turn, organised by the official bullets: structured, unstructured and semi-structured data, and data location.

Discovery tools find sensitive data in three ways, and each examines something different, costs something different and fails in a different way. Mature programmes layer them rather than relying on one.

A sensible plan at scale combines them. Labels are worth exactly as much as the process that applied them: markings from a recent, well-run project carry real evidence, while anything unmarked still has to be found. Metadata is good at deciding the order of work (a finance owner's folders before a build server's cache), even though it cannot settle what a file holds. The principle is to spend the expensive method where the cheap methods are least trustworthy or the stakes are highest, and never to let a cheap signal stand in for inspection where accuracy matters.

Structured data is the easiest type to run discovery against because it has a fixed schema: every row has the same typed columns, so a tool can read table and column names, data types and lengths from the database catalog before reading any values. Relational databases and data warehouses are the usual examples.

The schema is a starting point, not an answer. A catalog-only scan judges a column by its name and type, so it misses sensitive values that users typed into general-purpose text fields, legacy columns that were repurposed, and columns with cryptic or abbreviated names. However carefully name-based rules are refined, they still describe the label on the column, not what it holds. The fix is content inspection of stored values: sample or read the values in each column and test them with patterns and validation.

When a warehouse is too large to read in full on every scan cycle, reading every row is rarely affordable. The practical answer is statistical sampling of values: because a column tends to hold one kind of value, reading a modest number of its entries usually reveals what the whole column contains, and a misnamed column then gives itself away by its contents rather than its name. The catalog still tells the scanner which tables and columns exist, so the schema and the sampled values work together. Any marking applied to a table, whoever applied it, is only as reliable as the labelling method described above.

Unstructured data (documents, presentations, PDFs, images, audio and video) has no schema to read, so discovery has to analyse the content itself. Spreadsheets sit on the boundary: a file share sees them as files with no catalog schema, yet a header row gives their contents some structure, which is why some sources call them semi-structured and why a scanner should use the header as context when it reads them. The CBK names four content analysis techniques (pattern matching, lexical analysis, statistical analysis and hashing); the right one depends on what you are looking for. Optical character recognition is listed with them below because scanned content needs it, but it is a preprocessing step that turns images into text for those techniques, not a technique of its own.

Two distinctions are worth holding firmly. A whole-file hash finds exact copies only; change one byte and the hash differs, so it cannot find documents that merely discuss a subject. Pattern matching needs a fixed format; when documents share no code name or number format, lexical analysis is the tool. And a photographed or faxed page stored as an image contains pixels, not text: regular expressions and check digits find nothing until OCR has extracted the text. File metadata (author, owner, folder, dates) describes a file but says nothing reliable about what it contains.

Semi-structured data carries self-describing tags or keys inside each record but has no fixed schema shared by all records. JSON and XML documents are the standard examples, and email is semi-structured too: tagged header fields (From, To, Subject, Date) wrap a free-text body. Logs written as JSON, where each service adds its own fields, are a common cloud case.

Discovery cannot read one schema from a catalog, because records differ. What it can use is the structure each record carries: a parser walks the record's keys, and the name and position of a key (its key path in nested data) tells the scanner what kind of value to expect there. A number under shipment.tracking and the same digits under payer.pan deserve different judgements. Free-text values inside a record, such as an email body or a notes field, are then analysed with the unstructured techniques above.

Key context is also the main lever against false positives. In telemetry or booking records, for example, long numeric values are often device serial numbers or booking references that happen to share the length of a card number. The sound approach keeps every value in scope and judges each one better: use the value's key path as context, and validate candidates with the Luhn check that genuine card numbers satisfy, which most look-alike numbers fail. A fix that cuts alerts by taking data out of inspection, or by falling back on field names alone, reduces noise only by not looking.

A discovery result is only as good as the rules, the coverage and the readability of the data behind it: rule tuning decides false positives and false negatives, sampling decides what the scan can prove, and encryption decides what it can read at all.

False positives, false negatives and sampling

Every content rule trades false positives (benign data flagged as sensitive) against false negatives (sensitive data missed). A rule that matches any ten-digit number floods reviewers with phone numbers and order references; a rule tightened carelessly goes quiet and misses real data. Good tuning adds context and validation without discarding real matches.

• Validation: check digits (Luhn for card numbers) and format rules (for example, number ranges a national ID scheme never issues) reject look-alikes.
• Context terms: nearby words make a match more credible, but the list must be broad. Requiring one exact term such as passport beside a number creates false negatives wherever the same identifier sits under another label, such as a column headed Travel document. Proximity windows are a particular trap in spreadsheets and tables: the label sits once in a header row while the values sit in rows below it, often far beyond a short character window from the label. A sudden drop in alerts after tuning should prompt a check for misses, not just celebration.
• Weak fixes: any change that lowers alert volume by narrowing what is inspected, rather than by judging each match more accurately, trades false positives for false negatives.

Sampling controls cost but changes what a scan can claim. Inspecting a small random fraction of objects shows the typical content of a store and is good for triage and prioritisation. It cannot prove that sensitive objects are absent: rare objects are likely to fall outside the sample. Sampling rows within a structured column works better than sampling objects in a bucket, because a column is homogeneous while a bucket can hold one sensitive file among millions.

Encryption and protected content

Discovery can only inspect content it can read. Whether encryption blocks it depends on where decryption happens relative to the scanner.

The interpretation matters. A scan that reports "no sensitive data" in password-protected archives or client-side encrypted fields has not shown that the content is harmless; their sensitivity is unknown. Record such items as uninspected and resolve them another way: ask the data owner, or inspect at the point where the application decrypts. An honest report separates what was inspected from what could not be.

The service model decides what the customer can reach to run discovery, following the shared responsibility split.

• IaaS: the customer controls its virtual machines, volumes and storage, so it can run its own discovery agents and scanners against them.
• PaaS: the customer reaches its data through the platform's data interfaces and the access the service exposes; it does not reach the underlying hosts.
• SaaS: the customer has no access to the provider's servers or storage volumes. Discovery runs through the provider's APIs or export features, within what the contract grants. Infrastructure-level methods that work in IaaS are not part of the normal SaaS arrangement.

Moving data from IaaS to SaaS therefore changes how discovery is done, not who is responsible for it. The customer remains the data owner and still has to know where its sensitive data is and what it contains; the provider does not take over discovery and classification of customer data, and assurance about the provider's own controls says nothing about what the customer's data contains.

Because SaaS access is contractual, it has to be secured before signing: a provider whose agreement or interfaces later turn out not to allow regular, complete reads of the customer's own records leaves discovery with no route in, and that is far harder to negotiate after the data has moved. Check the access terms, the API scope and any rate limits against the scanning the programme plans to run.

Data location records every place a copy of the data exists, and it matters because the jurisdiction of storage, and of access, decides which laws, transfer rules and residency commitments apply.

Copies are easy to miss. An inventory that lists the primary database and its replicas is incomplete if it omits:

• Backups and snapshots, which may be kept in a different Region from the primary. Some providers retain automated backups for a period after a database instance is deleted, and manual snapshots persist until someone deletes them, so copies can outlive the instance they came from.
• Exports and extracts, such as extracts loaded into a reporting or machine-learning environment hosted elsewhere.
• Logs and telemetry that capture personal data (emails, IP addresses, identifiers) and are shipped to log or analytics services elsewhere.
• Caches, staging areas, test copies and archives.

The test for the inventory is whether something holds the records themselves. Systems that only process, route or protect the data without storing a copy raise other questions (where processing happens, who holds the keys) but are not locations of the data set.

Three consequences follow:

1. Each copy's location counts. A residency requirement is not met because the primary store is in the right place if backups replicate abroad or logs carrying personal data go to a foreign service. Operational data that contains personal data is still personal data.
2. Encryption does not change location. Ciphertext kept abroad is still data stored abroad; a law that requires in-country storage is about where the records sit, and keeping the keys at home does not change that. Encryption reduces exposure; it does not move the data.
3. Access location matters too. When people in another country, such as a provider's support engineers or another group company abroad, can reach data hosted at home, that access can bring the other country's laws and transfer rules into play, so data location records should capture where data can be accessed from, not only where it is stored. Under GDPR, for example, the EDPB's Guidelines 05/2021 treat remote access from a third country to data held in the EEA as a transfer when the party accessing it is a separate controller or processor, such as a provider's support staff. An employee of the same EEA controller working from abroad is not, by that alone, a transfer.

Moving data for residency is a location question, so it is settled by evidence of location: the copies left in the source Region (snapshots, exports, retained backups) survive the deletion of the instance they came from, and only discovery shows whether any remain. How data is then destroyed belongs to sub-domain 2.7.

Data that staff put into unapproved cloud services (shadow IT) is still the organisation's data, and finding it is part of knowing data location. A cloud access security broker (CASB) is the usual tool, and its deployment modes see different things.

The choice follows the question. To learn which services data is going to before you know what they are, analyse the logs you already collect: it needs no change to user traffic and covers every service that appears in them. To find sensitive files already stored in an approved tenant, however they got there, connect to that tenant in API mode.

In cloud environments, teams create buckets, databases and SaaS workspaces on demand, so a discovery inventory starts going stale the day it is finished. Discovery has to be a continuous, change-driven process rather than an annual project. The measure of a design is how quickly a new store or object is inspected after it appears, without relying on anyone remembering to ask for it.

When full scans grow too slow and costly, the efficient design is event-driven scanning: trigger discovery from resource creation and object write events so new data is inspected soon after it appears, plus a periodic sweep to catch anything events missed. This keeps both coverage and freshness: every new store and object is in scope, and none waits for the next scheduled run.

The discovery tool itself is a high-value asset. It needs broad access to work, and its findings concentrate sensitive information in one place. Two controls address the two risks:

• Least privilege for the scanner: read-only access to the stores it scans. Write or administrator rights (for example to quarantine files) widen the damage if its identity is compromised; remediation can be a separate, controlled workflow.
• Protect the findings as sensitive data: reports and dashboards that show matched values reproduce the data they found. Mask matched values so findings show location, data type and count rather than the values, restrict who can view them, and do not store them with ordinary operational logs. In short, findings deserve the same protection as the data they describe.