Data in Digital Transformation: Databases, Lakes, Warehouses and Governance (Cloud Digital Leader)

Data is valuable because it turns what happens in a business into evidence that people and systems can act on. The Cloud Digital Leader exam guide gives examples such as real-time business insights, identifying trends, informing strategic decision making and fueling AI.

• Real-time insights. When data is analysed as it arrives, leaders can see what is happening while they can still change it. Examples are demand that shifts during the day, a fraud attempt in progress, or a delivery running late. A monthly report describes last month. It cannot help a store manager move staff this afternoon.
• Identifying trends. Historical records show patterns over time, such as seasonal peaks, slow-moving products and changing customer habits. A company can then plan capacity, stock or staffing before the peak arrives.
• Informing strategic decisions. Data-driven decision making bases pricing, routing, investment and product choices on measured evidence rather than instinct.
• Fueling AI. Machine learning models learn patterns from examples, so the data is the raw material. A fraud model learns from past claims, including known fraud cases. A personalisation model learns from purchase, support and marketing history.

Data that is only stored creates little value. Keeping records safe, cheap or highly available protects the way they are already used, but it does not tell anyone something new. The value comes from analysing the data and acting on what it shows, for example using years of service records to predict which equipment will fail next.

Why AI needs connected data

An AI model is only as good as the data it learns from. When information about the same customers, products or assets is split across systems that never share records, every model sees only a slice of the picture, and its predictions are weaker. The cure is to connect the data so that each record can be joined to the rest. Adding volume or computing power to a fragmented picture leaves it fragmented. Machine learning also differs from hand-written rules: it learns patterns from examples, including patterns nobody thought to write down, which is why the quality and completeness of those examples matter so much.

A database runs the business's day-to-day transactions. A data warehouse holds cleaned, structured data from many systems for analysis. A data lake stores raw data of any type in its original format until someone decides how to use it. The three differ in purpose, in the kind of data they hold, and in when that data is given a structure.

Two related terms often appear beside these. A data mart is a subject-specific store, usually a subset of a warehouse, prepared for one team, so it holds curated structured data, not raw files. A dashboard or report is an output of business intelligence tools. It displays data to people but is not where the data is stored or updated.

Why analytics should not run on the transactional database

Transactional work and analytical work are different workloads. A transactional database is tuned for many small, fast reads and writes; analytical queries scan large volumes of history. Running heavy analysis on the system that serves customers makes the two compete, and day-to-day operations slow down. Analysis also usually needs data from several systems at once, which no single application database holds. A data warehouse addresses both points: it separates analysis from operations and consolidates cleaned data from every source in a store built for analytical queries.

Matching each need to a system

Many organizations need all three, each for a different job. A city transit operator, for example, records every fare payment as it happens (transactional work for a database). It keeps raw CCTV footage and vehicle sensor logs for future data science (varied, largely unstructured data for a lake). It reports ridership trends across every line and year (structured analysis across sources for a warehouse). Decide by the job each piece of data does, not by where the organization already keeps it. Which Google Cloud product fills each role (for example Cloud SQL, Spanner, BigQuery or Cloud Storage) is covered in the next lesson, on Google Cloud data management products.

A data lake still needs governance, because the flexibility of storing anything raw is also its main risk. If nobody records what each dataset is, who owns it, whether it is current and whether it contains personal information, the lake becomes a data swamp. The data is still there, but nobody can trust or safely use it.

The fix is to catalogue the lake, not to change what a lake is:

• A catalogue with metadata describes each dataset: its source, its contents, when it was last updated and what it is fit for.
• Named owners are accountable for each dataset's quality and for deciding who may use it.
• Access policies control who may see sensitive files, so compliance teams can show where personal data sits and who can reach it.

Governance keeps what makes a lake useful. Data can still land raw and in any format; what changes is that every dataset is described, owned and protected. Forcing everything into a fixed schema before loading would be schema-on-write, which is how a warehouse works, and it would still leave the questions of ownership and sensitivity unanswered.

Data is classified by source according to who collected it, not who it describes. First-party data is collected by the organization itself. Second-party data is another organization's first-party data shared directly with you. Third-party data is compiled by an outside provider and sold or licensed.

The common trap is a partner sharing data about people who are also your customers. That data is still second-party, because your organization did not collect it. Second-party and third-party data both come from outside, but they differ in a clear way. Second-party data is one partner's own records shared directly. Third-party data is a provider's product, compiled from many sources.

You may also meet the marketing term zero-party data. It means information a customer deliberately volunteers, such as stated preferences in a survey. It is not one of the categories the exam guide lists. Observed behaviour, such as purchase history an app records, is first-party data, not zero-party.

First-party data is usually the most valuable and the most trusted, because the organization knows how it was gathered and has a direct relationship with the customer. Second- and third-party data can add reach and context, but they describe the wider market or a partner's view, not what is happening in your own business right now.

Data is also classified by how it is organised. Structured data follows a fixed schema of rows and columns. Semi-structured data labels its values with field names or tags, but records need not share the same fields. Unstructured data has no predefined data model at all.

Two distinctions decide most cases. Named fields rule out unstructured. A fixed schema shared by every record is what makes data structured. JSON can look orderly, but if different records carry different fields it is semi-structured. Metadata is a separate idea: it is data about other data, such as a table's column names or a file's creation date, not the business content itself.

The type matters because it shapes where data is kept and how it is analysed. Structured data suits databases and warehouses. A lake can hold all three types. Unstructured data such as images, audio and documents is a large and growing share of what organizations hold, and modern AI can now draw insight from it.

An organization's data supply chain is the path data travels from the moment it is created to the moment it drives a business action. It has six stages: genesis, collection, processing, storage, analysis and activation.

Genesis and collection are easy to confuse. Genesis is the moment the data first exists. Collection follows it and gathers data that already exists from where it was created.

Fix problems at the stage that causes them

Thinking in stages helps locate a problem, and a problem is cheapest to fix at the stage whose job it is. Inconsistent formats, duplicates and missing values are processing problems, because processing is the stage that cleans, standardises and de-duplicates. Fixing them once there means every report, application and model downstream receives the same consistent data. Patching the same fault separately wherever it shows up later repeats the logic in many places, lets the copies drift apart, and leaves any consumer nobody patched still receiving bad data. Equally, data that is created correctly at genesis is not at fault simply because it later arrives in more than one format.

Data activation is the final stage of the data supply chain: it puts insights into the business workflows, applications and decisions where people or systems act on them. Without activation, accurate analysis creates no business value.

A common failure is good analysis that never reaches a decision. Imagine a logistics firm whose model accurately predicts which deliveries will run late, but the predictions sit in a report the dispatch team seldom opens, so late deliveries continue. The analysis is not the weak link; the missing step is getting the insight to the point of action. Activation closes that gap by delivering the insight inside the tools and processes people already use, for example raising an alert in the dispatch app so a driver can be rerouted, or triggering an automatic message to the customer.

Analysis and activation sit next to each other, so keep them apart. Analysis produces the insight, such as a score, a forecast or a segment. Activation applies it, such as an offer sent, a price changed, stock moved or an alert raised. Embedding live data and dashboards into everyday workflows with business intelligence tools is covered in the lesson on smart analytics and Looker.

Data governance is the set of agreed policies, roles and processes that decide who owns data, how it is defined, what quality it must meet, who may access it, and how its use is monitored. It matters because data that is inconsistent, insecure or non-compliant cannot be trusted for decisions or AI, and misusing it exposes the organization to regulatory and reputational risk.

A sound governance process runs in order and keeps running:

1. Discover and classify data up front, identifying sensitive fields such as personal information before anyone is given access.
2. Assign owners who are accountable for each dataset's quality and use.
3. Agree definitions and quality standards, so that terms like "customer" mean the same thing in every department.
4. Control access by role (least privilege), restricting sensitive fields to the people whose work needs them.
5. Monitor and audit continuously, so the organization can show regulators and auditors how data has been used over time, not just on the day the system launched.

The order matters, and so does completeness. You cannot protect data you have not yet identified as sensitive, access rules mean little without owners to decide them, and controls that nobody monitors cannot later prove how data was used. A program that skips a step leaves exactly that question unanswered.

Governance is about rules and responsibilities, not about a particular storage system. Where data physically sits, or how many copies exist, does not by itself decide who owns it, what its terms mean or who may see it. Good governance usually supports sharing, because people can use data confidently once they know it is accurate and they are allowed to.

Openness and interoperability are critical because they let data be shared, combined and moved across tools, teams and cloud providers. That removes data silos without creating dependence on one vendor (vendor lock-in). An open data platform rests on two things:

• Open standards and open-source formats. Many tools and providers can read data stored in an open file or table format (for example Apache Parquet, or an open table format such as Apache Iceberg). The data outlives any single tool, and changing tools later does not mean converting everything.
• Standard interfaces. SQL and open APIs let different tools query and exchange the same data, so teams can keep their preferred tools while working from one consistent source.

The benefit of openness is portability and choice: the organization, not a vendor, decides which tools work on its data and can change that decision later. It works alongside governance rather than replacing it.

Ending silos without lock-in

Silos can be removed in more than one way, so the deciding question is what each way costs in future flexibility. Consolidating every team onto one vendor\'s proprietary platform connects the data, but makes the whole organization depend on that vendor. Open formats and standard interfaces connect the data too, while keeping each team free to choose its tools and the organization free to change providers. When a requirement says "without tying the company to one vendor", that difference decides it.

Multicloud: query data where it lives

Some data cannot move. Regulation may require it to stay on a particular provider or in a particular region, or copying it may be too slow or costly. A multicloud analytics approach queries data in place across providers and gives a unified view without copying it. Google Cloud's examples include BigQuery Omni, which queries data held on other clouds, and BigLake, which manages tables over open formats such as Apache Iceberg. Querying in place and open formats solve different problems. Querying in place deals with where the data must stay; open table formats deal with who can read it later. A requirement that covers both needs both.