Smart Analytics on Google Cloud: Looker, Streaming Analytics, Pub/Sub, Dataflow and Spark (Cloud Digital Leader)

Looker democratizes data by letting business users explore governed data and build their own reports in a web browser, without writing SQL or waiting for an analyst. A sales manager picks fields such as region, product and revenue, applies filters, and Looker writes and runs the query for them.

Before self-service tools, every new question went into an analyst's queue. A manager who wanted last quarter's sales by region might wait days for a query to be written. That delay means decisions are made on instinct or on stale numbers. With Looker, the people closest to the decision can answer most of their own questions, and analysts spend their time on harder problems.

Self-service still has guardrails

Democratized does not mean uncontrolled. Looker access is governed by permissions:

• Role-based access decides what each user can do, such as view dashboards, explore data or edit the model.
• Data-level controls can limit which rows each user sees. A franchise chain, for example, can give every store manager the same sales model while each manager sees only their own store's figures.

So the goal is wider access to trusted data, within the rules the organization sets: more people answering their own questions, from one governed source, with permissions decided centrally.

Looker's semantic model is a shared layer, written in Looker's modeling language LookML, that defines each business metric and relationship once so every report uses the same definition. Data teams describe in LookML how tables join and how measures such as revenue, gross margin or active customer are calculated. Business users then explore through that model rather than writing their own calculations.

The business benefit is consistency. When a metric's logic is scattered across many separate queries and workbooks, two reports can disagree about the same figure and nobody can say which is right. With a metric defined once in the model, every dashboard and every self-service exploration reuses the same trusted logic, and a change to the definition updates every report that uses it.

The model is what lets governance and self-service coexist. When people's figures disagree, the lasting fix is to settle the business rule once in the model (for example, whether paused subscriptions count as active customers) so that every exploration inherits it, while users keep building their own views.

Analyzing and visualizing BigQuery data in Looker gives reports and dashboards that reflect the data in the warehouse now, because Looker queries BigQuery in place rather than working from a copy. Looker generates SQL from the model and sends it to BigQuery, which does the heavy processing; there is no nightly extract into a separate reporting store to go stale.

Looker does cache query results to avoid repeating identical work. How long a cached result is reused is set in the model, for example with a short maximum cache age or a datagroup tied to when new data is loaded. With a suitable policy, a dashboard reflects new rows shortly after they reach BigQuery. When the data itself arrives continuously (see the pipeline sections below), managers can watch near real-time dashboards, such as call-centre queue lengths updating within minutes.

The value for the business is the combination of BigQuery's scale with Looker's governed self-service: large datasets can be analysed interactively by many people, and everyone sees the same current figures.

Integrating data into workflows means putting Looker data where people already work, or into the systems that act on it, so an insight leads to action without anyone copying numbers by hand. Looker supports this in several ways:

• Scheduled delivery. Dashboards and reports can be sent on a schedule, or when data meets a condition, to destinations such as email, chat tools and file storage. Managers receive the report instead of having to remember to open it.
• Sending data to other services. Looker can send query results to other applications through integrations (Looker actions) and webhooks. For example, a list of overdue invoices can be sent to the finance team's collections tool, or a low-stock result can open a purchase request in the procurement system.
• Embedded analytics. Looker dashboards and visualizations can be embedded inside a company's own web application or portal, so customers or partners get analytics as part of the product they already use. A freight broker, for example, can show each shipper its own on-time delivery metrics inside the broker's customer portal.

The value is shorter time from insight to action. Every manual hand-off between seeing a number and acting on it adds delay and room for error; workflow integration moves the data to the people and systems that act on it, in the form they need.

Real-time streaming analytics is critical because many business decisions lose their value if they are made minutes or hours after the event. Streaming analytics processes data continuously as it is generated, so a business can act while the moment is still happening. Batch analytics collects data and processes it at intervals, such as nightly, which suits reviews of a closed period.

The decision rule is about time to act: if the insight is worthless a few minutes later, it needs streaming. If it reviews a closed period, batch is cheaper and simpler. Modern businesses usually need both, which is why a pipeline that can handle both styles matters (see Dataflow below). Streaming also feeds live dashboards and AI models that score each event, such as an anomaly score for each sensor reading.

Pub/Sub is Google Cloud's fully managed, scalable messaging service: it receives event messages from producers and delivers them asynchronously to every system that subscribes. It is typically the front door of a streaming pipeline, absorbing large volumes of events, such as millions of player actions or sensor readings per second, from apps and devices.

• Publishers send messages to a topic.
• Each consuming system has its own subscription to the topic and receives the messages independently.

The key business value is decoupling. A producer publishes each event once and moves on; it does not need to know who consumes the event, how many consumers there are, or whether one of them is running slowly. Pub/Sub holds messages until each subscriber acknowledges them, so a consumer that falls behind catches up later without holding anyone else back. Teams can add consumers over time by adding subscriptions, and producers and consumers can be changed and scaled independently.

Pub/Sub is not a processing engine. Apart from light per-message changes (single message transforms, such as masking or reformatting a field), it does not de-duplicate, join or aggregate events, and it does not visualize anything. It can deliver messages straight into BigQuery through a BigQuery subscription, which writes each message as it arrives. When events must be cleaned, combined or summarised first, a processing step such as Dataflow sits between Pub/Sub and the warehouse.

Dataflow is Google Cloud's serverless service for processing data in both batch and streaming mode, running pipelines written with the open-source Apache Beam programming model. Dataflow transforms data on the way through a pipeline: cleaning, enriching, de-duplicating, joining and aggregating it, for example joining a stream of sensor readings with a table of device details and averaging the readings over five-minute windows before they are stored.

Two things make Dataflow valuable to a business:

• One programming model for batch and streaming. The same Beam pipeline code can process archived files and a live stream. Logic is written once, so a backfill over last year's history and today's live feed calculate results the same way, and there is one codebase to maintain instead of two.
• Serverless operation. There are no clusters to size or manage; Dataflow provisions and scales workers automatically and charges for the resources a job uses. That suits small data teams that want results, not infrastructure.

Dataflow is the natural choice for new pipelines. Existing Spark or Hadoop jobs would have to be rewritten as Beam code to run on Dataflow, which is where the next product fits.

Managed Service for Apache Spark (formerly Dataproc) runs open-source Apache Spark and Hadoop-ecosystem jobs on managed Google Cloud clusters, so an organization can move existing data processing jobs to the cloud and generally run them as they are. It is the modernization path when the constraint is "keep the code we have".

• Keep jobs and skills. The service runs the same open-source frameworks the jobs already use, so code, libraries and team skills carry over.
• Elastic, pay-for-use clusters. A cluster can be created in minutes for a workload, resized as demand changes, and deleted once the work is done, so you pay while jobs run rather than for hardware that stands waiting.
• Less operations work. Google Cloud manages cluster provisioning and the software stack, instead of the company maintaining its own hardware.

Google Cloud also offers a separate product, Serverless for Apache Spark, which runs Spark workloads without the customer managing clusters at all. Spark on Google Cloud therefore does not always mean operating clusters, but the managed-cluster service is the one built for lifting existing Spark and Hadoop estates.

A modern streaming pipeline on Google Cloud typically runs Pub/Sub → Dataflow → BigQuery → Looker: Pub/Sub ingests events, Dataflow processes them continuously, BigQuery stores and analyses the results, and Looker turns them into governed dashboards and workflow actions. Every stage is managed, and the ingest, processing and warehouse stages are serverless.

Real scenarios add constraints, and the right design is the one that meets all of them, not just most:

• Latency. A pipeline is only as fast as its slowest stage. If results must appear within a minute or two, every stage has to work continuously; one scheduled step sets the delay for the whole pipeline.
• Operations. When the team wants nothing to run or patch, check every stage for clusters, not only the processing step.
• Existing code. Where a large body of working Spark or Hadoop jobs exists, the cost and risk of rewriting usually outweigh the benefit, so move those jobs as they are.

Organizations often mix approaches. A manufacturer, for example, might move its nightly Spark reporting jobs to Managed Service for Apache Spark as they are, while building a new serverless feed on Pub/Sub and Dataflow that flags equipment faults as readings arrive. Choosing per workload beats forcing every job onto one product. Which database or storage product holds the data is covered in the companion lesson on Google Cloud data management products.