Cloud Digital Leader: Modern Operations, Observability, Resilience and SRE

Modern cloud operations means running workloads with managed tools, automation and measured reliability targets instead of hand-built scripts and reactive firefighting. On premises, an operations team often installs its own monitoring agents, writes scripts to check servers and learns about problems when users call. In the cloud, the provider runs the monitoring platform as a managed service, data from every service lands in one place, and the team decides in advance how reliable each service needs to be.

Three ideas run through this whole topic:

• See what is happening — collect metrics, logs, traces, profiles and errors so problems are spotted, ideally before customers notice (observability).
• Design so failures do not become outages — redundancy, replication, autoscaling and backups (resilience).
• Agree how reliable is reliable enough — measure the service with indicators, set objectives on them and balance reliability against the speed of change (DevOps and Site Reliability Engineering).

Controlling what all of this costs — budgets, quotas, billing reports and the resource hierarchy — is task 6.1 and has its own lesson.

Google Cloud Observability is Google Cloud's managed suite for watching application health and performance; it was previously called the operations suite (and before that Stackdriver). It collects the main kinds of operational signal — metrics, logs, traces, code profiles and errors — from Google Cloud services and from applications, so teams can detect, investigate and fix problems from one place. Each product answers a different question, and telling them apart is the core of this bullet.

What moving to Observability changes is visibility: signals from every service arrive in one managed place, dashboards and alerts are configured rather than scripted, and teams can spot and investigate problems sooner. Deciding what to do about a problem, and doing it, remains the team's job. Observability is about application health and performance; detecting security threats is a separate set of tools covered in the security domain.

The right product is the one whose kind of signal matches the problem in the scenario. These near-twin pairs are where people slip:

• Monitoring vs Logging. A threshold on a number (memory use above a set level, request latency over a limit) is a metric condition in Cloud Monitoring. A searchable record of what each service wrote during an incident is Cloud Logging.
• Trace vs Profiler. Both help with "slow", at different levels. Cloud Trace looks across services at the timeline of one request. Cloud Profiler looks inside one service's code at which functions consume CPU and memory — useful for cutting compute cost, and safe to leave running in production because of its low overhead. Cloud Monitoring can show a VM's total CPU use but not which code causes it.
• Error Reporting vs Logging vs Monitoring. Exceptions appear as individual entries in Cloud Logging, and Cloud Monitoring can alert on an error rate, but only Error Reporting groups exceptions by cause into error groups, with counts and first-seen times, and notifies the team when a new group appears.
• Checking from the outside. Metrics, logs, traces and errors are all produced by requests that reach the application; a request that never arrives — because of a DNS, network or load-balancer problem — leaves no trace inside. A Cloud Monitoring uptime check sends requests to a public URL from several locations on the internet, as a visitor would, and an alerting policy on it notifies the team when those requests fail.

These are three related but different cloud operations terms: operational excellence describes how a team runs its workloads, while reliability and high availability describe what the workload achieves.

Two recovery measures are worth knowing by name: the recovery point objective (RPO) is how much recent data the business can afford to lose, and the recovery time objective (RTO) is how long it can afford to be down.

Resilient infrastructure keeps serving when a component fails, and the main technique is redundancy: running more than one copy of each part in separate failure domains, so no single failure takes the service down. In Google Cloud the failure domains are nested: a zone is a deployment area inside a region, and a region is a separate geographic location with several zones.

The rule is that redundancy protects only against failures of a domain the copies do not share:

Resilience comes from where the copies are placed, not from how large each machine is.

Replication keeps copies of data in more than one zone or region so that the redundant compute has current data to serve from. How far behind the copy runs sets the RPO: continuous replication keeps it seconds or minutes behind, while a copy refreshed once a night can be up to a day behind. For example, an order system that may lose at most the last five minutes of orders needs continuous replication to wherever it fails over.

Scalable infrastructure adds and removes capacity as demand changes, and backups preserve earlier copies of data; both are part of resilient design, and each protects against a different risk.

Scaling. A managed instance group runs identical virtual machines from a template, can spread them across zones, recreates failed instances (and, with a health check, unhealthy ones) automatically and, with autoscaling, adds instances as load rises and removes them when it falls. For demand that rises and falls — a tax-filing site busy near the deadline, a retailer during a holiday sale — capacity follows the load, so the service stays responsive and the company pays for extra instances only while they are needed. Serverless products such as Cloud Run scale in the same spirit without managing VMs at all.

Backups vs replication. Replicas are kept in sync, so they copy mistakes as faithfully as good data: a mistaken delete, a corrupted table or a faulty batch job reaches every replica within seconds. A backup holds the data as it was at an earlier moment, which is what lets the team undo a bad write. To be useful during a regional outage, backups must be stored outside the region they protect.

Resilience is about processes too: tested restore procedures, automated deployments that can be rolled back, and regular practice of failover, so recovery works when it is needed.

The four golden signals are latency, traffic, errors and saturation — the measurements Google's Site Reliability Engineering practice recommends watching for any user-facing system, because together they show whether it is serving users well.

Saturation is the early warning: a message queue whose worker pool is almost fully busy may still process every message on time, and only once the pool is exhausted do latency and errors climb. The golden signals are the measurements; broader goals such as availability are calculated from them. In Google Cloud these signals are collected as metrics and charted and alerted on in Cloud Monitoring.

An SLI is the measurement, an SLO is the internal target set on that measurement, and an SLA is the contract with customers that says what happens if a promised level is missed. DevOps is the broad culture of developers and operators sharing responsibility for delivering and running software; Site Reliability Engineering (SRE) is Google's concrete way of practising it, built on these three terms.

The SLO is normally set stricter than the SLA. Missing the SLO then warns the team while there is still time to act before the contract is breached. Reading a result works in two steps: name each figure, then compare. Suppose an SLA promises 99%, the SLO is 99.5% and this month's SLI is 98.8%. The measurement is below both figures, so the SLO is missed (the team must act) and the SLA is breached too, so customers are owed the remedy the contract states. Compare the SLI with each target separately; missing one does not imply missing the other.

An error budget is the amount of unreliability an SLO allows — for a 99.9% SLO, the remaining 0.1% of requests or time — and it is how SRE balances the speed of change against reliability. While budget remains, the team can take risks: ship features, run experiments, release often. When outages have used up the budget, the team pauses risky releases and puts its effort into reliability work until the budget recovers. The budget is a shared, agreed rule, which is what lets developers and operators settle release decisions with data instead of argument.

Other DevOps and SRE practices that support reliability:

• Blameless postmortems — after an incident, the team writes up what happened, why and how to prevent it, focusing on systems and processes rather than blaming individuals, so people report problems openly.
• Reducing toil through automation — repetitive manual operational work is automated so engineers can spend time on improvements.
• Small, frequent, gradual releases — changes rolled out in small steps (for example to a fraction of users first) are easier to observe and roll back.
• Monitoring and alerting on symptoms users feel — alerts tied to SLOs and the golden signals rather than to every internal event.