Cloud Monitoring and Logging: Alerts, Log Sinks, and Audit Logs

The distinction that unlocks this whole topic: a metric is a numeric measurement sampled over time — CPU utilization, request count, queue depth — stored as a time series and handled by Cloud Monitoring; a log is a discrete, timestamped record of something that happened — a request line, an error with a stack trace, an audit event — handled by Cloud Logging. Metrics answer "how much, how fast, how many"; logs answer "what exactly happened and in what order."

The two connect in both directions: log-based metrics convert log entries into time series so Monitoring can chart and alert on them, and both feed the same incident workflow. Many services emit platform metrics and platform logs automatically with zero setup; agents and instrumentation extend coverage inside VMs and applications. "Alert when X exceeds a threshold" points at Monitoring; "find out which user deleted the bucket" points at Logging — specifically audit logs.

An alerting policy watches one or more conditions and opens an incident — and notifies you — when a condition is met. A metric-threshold condition names the resource type and metric (say, Compute Engine instance CPU utilization), a comparison and threshold (above 75 percent, say), and a retest window (duration): the condition must stay true for the whole window before an incident opens. That window is the tool for ignoring short spikes — set it to the length of time the problem must persist, because a shorter window notifies earlier than required and a longer one notifies late. Behind the chart, an alignment period and aligner (mean, max, rate, or a percentile such as the 95th for distribution metrics) reduce raw points into the series the condition evaluates.

Notifications go to notification channels — email, SMS, Slack, PagerDuty, webhooks, and Pub/Sub for programmatic handling — which you create once and attach to policies. Three other policy fields matter in practice:

• Documentation — free text (Markdown) included in every notification, used for triage steps and runbook links so the responder knows what to check first.
• Severity — a label (Critical, Error, Warning) carried on incidents and notifications so responders and routing can prioritize.
• Auto-close duration — how long an open incident stays open after metric data stops arriving before Monitoring closes it (default seven days). When data keeps arriving and the condition is no longer met, the incident closes on its own; auto-close only covers the case where the data disappears.

Create policies in the Console's Alerting page, via the Monitoring API, or with gcloud monitoring policies create and a JSON or YAML policy file; channels are managed with gcloud monitoring channels. Besides threshold conditions, metric-absence conditions fire when data stops arriving (a heartbeat pattern) and forecast conditions fire when a metric is predicted to breach soon.

Uptime checks, public and private

An uptime check tests whether an endpoint answers — HTTP, HTTPS or TCP — by probing it from several Google locations around the world, recording reachability and latency (not CPU or other resource metrics). Pair it with an alerting policy on the check's result; requiring failures from several checker locations before the condition is met keeps a single location's network blip from paging anyone. A private uptime check covers internal endpoints with no external IP: register the endpoint (a VM or internal load balancer's address) in Service Directory, create the check against that Service Directory service so it runs from Google's VPC checkers, and allow ingress from 35.199.192.0/19 in your VPC firewall. The endpoint never has to be exposed to the internet.

Watching many projects

Monitoring is provisioned per project, but a metrics scope lets one scoping project chart and alert on metrics from many monitored projects — dashboards and alerting policies in the scoping project read metrics across all of them while the projects themselves stay separate. And if the metric you need is not emitted by default (memory utilization on a VM is the classic case), the fix is an agent or a custom metric, covered next.

When platform metrics do not cover what you need — a business count such as sign-ups per hour, or a value held only inside the application's own state — you create and ingest custom (user-defined) metrics. Application code writes them through OpenTelemetry instrumentation exported to Cloud Monitoring (the recommended route) or directly through the Cloud Monitoring API with client libraries; the Ops Agent can also collect application metrics (Prometheus-style endpoints among them) from software running on a VM. Metrics written through the API use the custom.googleapis.com/ prefix, carry their own labels, and afterward behave like platform metrics: chartable in Metrics Explorer and usable in alerting policies. A value that never reaches a log can only arrive this way.

Log-based metrics are the second route, extracting numbers from logs you already have without touching code:

Define one from the Logs Explorer or with gcloud logging metrics create error-count --log-filter="severity>=ERROR", then build an alerting policy on it like any other metric. User-defined log-based metrics count only entries that arrive after the metric is created — they are not retroactive. Google also predefines system log-based metrics (such as log bytes ingested and entry counts per log) that exist without any setup.

Log-based metric or log-based alerting policy?

A log-based alerting policy (a log match condition) notifies when a single matching entry appears — "tell me every time this specific error is logged" — with a configurable minimum time between notifications. It does not count. When the requirement is a rate or a count over a window, or a percentile, use a log-based metric plus a metric-threshold policy instead. Rule of thumb: one event matters on its own → log match alert; how many or how slow over time → log-based metric. If the signal is in logs, both routes need zero code changes; if it is not, instrument a custom metric.

Cloud Audit Logs record who did what, where, and when across Google Cloud, in four types with different defaults:

"Which admin deleted the instance?" is answered by Admin Activity logs, which are present even if nobody ever configured logging. "Who read the data?" needs Data Access logs, which someone must have enabled before the reads happened.

Configuring Data Access audit logs

Data Access logging lives in the audit configuration of an IAM policy (Console: IAM and Admin, Audit Logs; or an auditConfigs block set with gcloud ... set-iam-policy). For each service you choose the log types — ADMIN_READ, DATA_READ, DATA_WRITE — and can list exempted principals whose calls generate no Data Access entries at all (typically a high-volume automation identity whose routine reads would flood the logs with noise). Audit configs set on an organization or folder are inherited by every project beneath it, including projects created later, and combine additively with the project's own settings — a child cannot switch off a log type its parent enabled. Contrast exemptions with exclusion filters: an exempted principal's entries are never generated, while an exclusion drops entries that were generated from storage.

VPC Flow Logs vs firewall rules logging

The implied rules (deny all ingress, allow all egress) cannot be logged. To record blocked inbound attempts, create an explicit ingress deny-all rule with logging enabled at priority 65534: because it sits below your allow rules but above the implied deny at 65535, it catches exactly the traffic nothing else allows — traffic behaviour does not change, and each blocked attempt is logged with that rule. Both network log sources land in Cloud Logging and can be routed anywhere a sink can point.

Every log entry a project receives passes through the Log Router, which compares it against the project's sinks and delivers copies to each matching destination. Storage lands in log buckets:

What drives the Logging bill

Cloud Logging charges for the volume of logs stored in log buckets, measured when they are ingested (with a monthly free allotment), plus a separate charge for retaining logs beyond the default 30 days. _Required is not charged. Consequences: an exclusion filter on the _Default sink that drops a noisy, worthless log is what removes its ingestion cost; shortening retention only trims the beyond-30-days charge (useful when retention exceeds what policy needs, useless for ingestion volume); and routing logs into an extra bucket adds stored volume rather than saving any. Excluded entries still pass through the Router and can be captured by other sinks — they are just not stored by that sink's destination.

Sinks and destinations

A sink is a filter plus a destination: another log bucket (including one in a central project), BigQuery for SQL analysis over exported tables, Cloud Storage for cheap long-term files, and Pub/Sub — the near-real-time streaming path to external systems, where an on-premises SIEM or a Dataflow pipeline consumes a subscription on the topic.

gcloud logging sinks create audit-to-bq \
  bigquery.googleapis.com/projects/sec-proj/datasets/audit_logs \
  --log-filter='logName:"cloudaudit.googleapis.com"'

A sink writes as its writer identity, a service account whose email the command returns. Whether you must grant it access depends on how and where:

• No grant needed when the destination is a log bucket in the same project.
• Granted for you when you create the sink in the Console and have Owner access on the destination.
• Grant it yourself when you use gcloud or the API, or the destination is in another project: BigQuery Data Editor on the dataset (Data Viewer can only read, so delivery fails), Pub/Sub Publisher on the topic, Storage Object Creator on the bucket, or Logs Bucket Writer on a log bucket in another project.

Without the grant the sink simply delivers nothing. Aggregated sinks created on a folder or the organization with --include-children route matching logs from every project underneath, including projects created later — a sink on a folder limits that to the projects in that one branch of the hierarchy, while an organization-level sink covers everything.

Sinks are not retroactive

A new sink only routes entries that arrive after it exists. To get entries that are already stored, use gcloud logging copy, which copies log entries from a log bucket to a Cloud Storage bucket (the source keeps its copy); the sink then handles everything new.

Observability Analytics — the feature formerly called Log Analytics; you will see both names — lets you run SQL, including joins and GROUP BY aggregations, over logs that stay in their log bucket. You upgrade a bucket to use it (new buckets can be created upgraded) and then query from the Observability Analytics page in the Console, with no sink and no copy of the data. The upgrade cannot be undone.

An upgraded bucket can also get a linked BigQuery dataset: a read-only link through which BigQuery reads the log data that Logging stores, so analysts can query logs with BigQuery tools and join them with existing BigQuery tables without exporting a second copy. Only an upgraded bucket can have a linked dataset; queries run through BigQuery are billed as BigQuery queries.

The Logs Explorer is where you read logs: pick resource type, log name, and severity from the menus, or write queries in the Logging query language — boolean expressions over entry fields, such as resource.type="cloud_run_revision" AND resource.labels.service_name="checkout" AND severity>=WARNING. Severity is ordered (DEFAULT, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY), so >=WARNING includes everything from WARNING up; : is a substring match (textPayload:"timeout") while = is an exact match. Time bounds come from the page's time-range control unless the query compares timestamp itself; a histogram above the results shows entry volume over that range, and a shared query link can carry the query and its pinned time range to a colleague.

Clicking an entry expands its full record: timestamp, severity, resource labels and the payload (text, JSON, or protocol buffer). For audit entries the fields that matter are protoPayload.authenticationInfo.principalEmail (who made the call), protoPayload.methodName (what operation), and protoPayload.resourceName (which resource). Useful day-2 moves: "Show entries with same" to pivot on a field, saved queries, live streaming while reproducing an issue, and creating a log-based metric or an exclusion straight from the current filter.

From the command line, gcloud logging read 'resource.type="gce_instance" AND severity>=ERROR' --limit=20 --freshness=1h returns matching entries and gcloud logging write injects a test entry. For aggregations the query language cannot express — say, ranking the source IPs that sent the most bytes in VPC Flow Logs — use Observability Analytics SQL.

The Ops Agent is the single agent for Compute Engine VMs that collects both logs and metrics; it replaces the legacy Logging and Monitoring agents, which are deprecated. Without it, Monitoring sees a VM only from the hypervisor: CPU, disk and network throughput. With it you get in-guest telemetry — memory utilization, swap, disk space used — plus system logs, and through /etc/google-cloud-ops-agent/config.yaml you add receivers for application log files (a files receiver pointing at the application's log path) and third-party applications, wired into pipelines. A missing memory metric or in-VM application log means "install and configure the Ops Agent."

Deploy it per VM with the install script, at fleet scale with an agent policy (VM Manager keeps it installed on VMs matching labels or zones), or bake it into images. The agent writes as the VM's attached service account, which needs Logs Writer (roles/logging.logWriter) and Monitoring Metric Writer (roles/monitoring.metricWriter), and the VM's access scopes must allow logging and monitoring writes. Similar-sounding roles do not help: Logs Bucket Writer is for a sink routing into a log bucket, Cloud Trace Agent writes trace spans, and Logs Viewer / Monitoring Viewer are read-only. Installation and permission are separate requirements: a running agent still delivers nothing its identity is not allowed to write.

The Ops Agent is not supported on GKE nodes. GKE collects system and workload logs and system metrics itself, and uses Managed Service for Prometheus for application metrics.

Managed Service for Prometheus

Google Cloud Managed Service for Prometheus stores Prometheus-format metrics in Google's global time-series backend (Monarch), so you keep PromQL, exporters, recording rules and dashboards while Google runs storage and scale. Metrics are queryable with PromQL in Cloud Monitoring and alertable like everything else. It offers two collection modes:

Recognizing which tool fits which symptom is usually the whole question:

Index advisor in Cloud SQL has prerequisites: an Enterprise Plus edition instance with Query Insights enabled, and it does not run on read replicas. Turning on Query Insights is not enough by itself: you also select the separate Enable index advisor option in the instance's Query Insights configuration, then review its recommendations.

These tools answer "is it Google or is it us", "what should we optimize", and "what is going on across this application".

• Personalized Service Health filters Google Cloud service disruptions down to the ones that touch your resources, rather than the public status dashboard's global view, and can send notifications as an event is posted and as it changes. It is the first stop for "is this a Google-side outage?"
• Active Assist is the family of recommenders that analyze observed usage and suggest changes: idle VM and idle disk recommendations, VM machine type (rightsizing) recommendations, unattended-project cleanup, and IAM role recommendations that shrink over-broad grants.
• Cloud Hub is the Console home for operating the applications you register in App Hub. Per application it gathers health signals (alerting incidents and key metrics), Google-side events such as service incidents and planned maintenance, and deployment status, so operators get an application-centric view out of the box rather than assembling one.
• Gemini Cloud Assist is the AI assistant for operating Google Cloud. In Monitoring and Logging it helps write queries and explain charts, and its investigations examine the live logs, metrics, errors and configuration around an incident, propose likely root causes, and take natural-language follow-up questions.

Do not confuse Gemini Cloud Assist with Gemini Code Assist, which helps developers with source code in the IDE: incident investigation over live telemetry is Cloud Assist.