Cloud Monitoring and Logging: Alerts, Log Sinks, and Audit Logs

The distinction that unlocks this whole topic: a metric is a numeric measurement sampled over time — CPU utilization, request count, queue depth — stored as a time series and handled by Cloud Monitoring; a log is a discrete, timestamped record of something that happened — a request line, an error with a stack trace, an audit event — handled by Cloud Logging. Metrics answer "how much, how fast, how many"; logs answer "what exactly happened and in what order."

The two connect in both directions: log-based metrics convert log entries into countable time series so Monitoring can chart and alert on them, and both feed the same incident workflow. Many services emit platform metrics and platform logs automatically with zero setup; agents and instrumentation extend coverage inside VMs and applications. On the exam, "alert when X exceeds a threshold" points at Monitoring; "find out which user deleted the bucket" points at Logging — specifically audit logs.

An alerting policy watches one or more conditions and opens an incident — and notifies you — when a condition is met. A metric-threshold condition names the resource type and metric (say, Compute Engine instance CPU utilization), a comparison and threshold (above 80 percent), and a retest window or duration (for 5 minutes) so momentary spikes do not page anyone. Behind the chart, an alignment period and aligner (mean, max, rate) reduce raw points into the series the condition evaluates.

Notifications go to notification channels — email, SMS, Slack, PagerDuty, webhooks, and Pub/Sub for programmatic handling — which you create once and attach to policies. Policies can also carry user-defined documentation, a runbook snippet included in every notification so the on-call engineer knows what to do at 3 a.m. Create policies in the Console's Alerting page, via the Monitoring API, or with gcloud monitoring policies create supplying a JSON or YAML policy file; channels are managed with gcloud monitoring channels. Besides threshold conditions, know that metric-absence conditions fire when data stops arriving (a heartbeat pattern) and forecast conditions fire when a metric is predicted to breach soon — and that uptime checks probe a public endpoint from multiple locations and pair naturally with an alerting policy on the check's result.

Two scope notes for the exam. Monitoring is provisioned per project, but a metrics scope lets one scoping project view and alert on metrics from many monitored projects — the standard multi-project ops pattern. And if the metric you need is not emitted by default (memory utilization on a VM is the classic case), the fix is an agent or a custom metric, covered next.

When platform metrics do not cover what you need — queue depth in your application, orders processed per minute — you create and ingest custom metrics. Application code writes them through the Cloud Monitoring API using client libraries or, preferably today, through OpenTelemetry instrumentation exported to Cloud Monitoring; the Ops Agent can also collect application metrics (Prometheus-style endpoints among them) from software running on a VM. Custom metrics are written under the custom.googleapis.com/ prefix, carry their own labels, and behave exactly like platform metrics afterward: chartable in Metrics Explorer, usable in alerting policies, subject to the same aligners.

Log-based metrics are the second route, extracting numbers from logs you already have instead of instrumenting code. A counter metric counts entries matching a filter — every log line with severity=ERROR from your service, say — while a distribution metric extracts a numeric value from matching entries (a latency field) and records its distribution. Define one from the Logs Explorer or with gcloud logging metrics create error-count --log-filter="severity>=ERROR", then build an alerting policy on it like any other metric.

Choosing between them is a standard exam decision: if the signal already appears in logs, a log-based metric gets you alerting with zero code changes; if you need a precise measurement the logs do not contain, instrument a custom metric. Google also predefines some system log-based metrics for common events, and user-defined log-based metrics are calculated only from logs that arrive after the metric is created — they are not retroactive, a subtle gotcha worth remembering.

Cloud Audit Logs record who did what, where, and when across Google Cloud, and the exam expects you to know the types and their defaults cold:

The classic question — "which admin deleted the instance?" — is answered by Admin Activity logs; "who read the data?" needs Data Access logs, which someone must have turned on beforehand (in the Console under IAM and Admin, Audit Logs, or via an audit configuration in policy). Admin Activity, System Event, and Access Transparency logs are retained in the non-configurable _Required bucket.

Two network log sources sit alongside audit logs. VPC Flow Logs are enabled per subnet (gcloud compute networks subnets update app-subnet --region=us-central1 --enable-flow-logs) and record a sample of network flows to and from VM instances — with configurable aggregation interval and sampling rate — for traffic analysis, forensics, and cost attribution. Firewall Rules Logging is enabled per firewall rule and records connections that the rule allowed or denied; the implied rules cannot log, so seeing denied traffic requires an explicit deny rule with logging on. Both land in Cloud Logging and can be routed anywhere a sink can point.

Every log entry a project receives passes through the Log Router, which compares it against the project's sinks and delivers copies to each matching destination. Storage lands in log buckets: _Required holds Admin Activity, System Event, and Access Transparency audit logs for a fixed retention you cannot change or disable, and _Default receives everything else via a default sink, with configurable retention. You can create additional user-defined log buckets — including in specific regions for data-residency needs — and upgrade any bucket to Log Analytics, which lets you query its logs with SQL and optionally link a BigQuery dataset over them.

A sink is a filter plus a destination. Destinations: another log bucket (including one in a central project for cross-project aggregation), BigQuery for SQL analysis over exported tables, Cloud Storage for cheap long-term archival, and Pub/Sub — the path to external systems, where a subscriber (or a connector like Dataflow) forwards entries to an on-premises SIEM or third-party tooling. That Pub/Sub answer is exactly what the exam wants for "export logs to an external or on-premises system."

gcloud logging sinks create audit-to-bq \
  bigquery.googleapis.com/projects/sec-proj/datasets/audit_logs \
  --log-filter='logName:"cloudaudit.googleapis.com"'

Creating a sink mints a writer identity — a service account whose email the command returns — and you must grant it write access on the destination (for example BigQuery Data Editor on the dataset) or the sink silently fails to deliver. Aggregated sinks created at the folder or organization level with --include-children export logs from every project underneath, the standard central-security pattern. Exclusion filters on a sink drop noisy entries before storage to control cost — excluded logs still transit the Router (and can be captured by other sinks) but are never stored by that sink's destination.

The Logs Explorer in the Console is where you read logs: pick resource type, log name, and severity from the menus, or write queries in the Logging query language — boolean expressions over entry fields, like resource.type="gce_instance" AND severity>=ERROR AND textPayload:"timeout". The histogram shows matching volume over time, and clicking an entry expands the full record: timestamp, severity, resource labels, and the payload (text, JSON, or protocol buffer). For audit entries, the details reveal the fields questions hinge on — protoPayload.authenticationInfo.principalEmail (who), protoPayload.methodName (what), and the resource acted upon.

Useful moves for day-2 work: pin a query's time range before sharing its URL with a teammate; use "Show entries with same" to pivot on a field; save frequent queries; and stream logs live while reproducing an issue. From the same interface you can create a log-based metric or an exclusion from the current filter — the query you debug with becomes the filter you alert or economize with.

From the command line, gcloud logging read 'resource.type="gce_instance" AND severity>=ERROR' --limit=20 --freshness=1h returns matching entries, and gcloud logging write can inject a test entry. Buckets upgraded to Log Analytics add a SQL surface for aggregations the query language cannot do — top talkers from VPC Flow Logs, error rates grouped by service — without exporting anything to BigQuery first.

The Ops Agent is the single, unified agent for Compute Engine VMs, combining logging and metrics collection (it replaced the separate legacy logging and monitoring agents). Without it, Monitoring sees a VM only from the hypervisor: CPU, disk and network throughput. With it, you get the in-guest telemetry exam questions probe for — memory utilization, swap, disk space used — plus system logs (syslog, Windows event log) and, via its YAML configuration at /etc/google-cloud-ops-agent/config.yaml, application logs and metrics from third-party software such as web servers and databases. Any question about a missing memory metric or in-VM application log is answered by "install the Ops Agent."

Deploy it per VM with the install script, at fleet scale with an agent policy (VM Manager applies and maintains the agent across VMs matching labels or zones), or bake it into golden images. The agent authenticates as the VM's attached service account, which needs the Logs Writer and Monitoring Metric Writer roles — a permissions detail that doubles as a troubleshooting answer when an installed agent ships nothing.

Google Cloud Managed Service for Prometheus is Monitoring's answer for teams standardized on Prometheus: it collects Prometheus-format metrics and stores them in Google's planet-scale time-series backend, so you keep PromQL, exporters, and dashboards while Google handles storage and scale. On GKE you enable managed collection (on by default in newer clusters) and declare scrape targets with PodMonitoring resources; self-deployed collection swaps your Prometheus binary for Google's drop-in replacement anywhere else. Collected metrics are queryable with PromQL right in Cloud Monitoring, alongside — and alertable like — everything else. Exam cue: "keep our Prometheus dashboards and PromQL but stop managing Prometheus storage" maps to Managed Service for Prometheus.

Beyond metrics and logs, the exam names a diagnostic toolkit, and recognizing which tool fits which symptom is usually the whole question. Cloud Trace is distributed tracing: it follows a request across services, breaking latency into spans, so "which service in the chain makes requests slow" is a Trace question. Cloud Profiler continuously samples CPU and memory usage of running applications with negligible overhead and shows flame graphs of which functions burn resources — "why is this service using so much CPU in production" is Profiler. Query Insights lives in Cloud SQL and AlloyDB and diagnoses database query performance, surfacing the slowest queries and their execution plans, with an index advisor recommending missing indexes.

The health and optimization layer: the Personalized Service Health dashboard shows Google Cloud service disruptions relevant to your specific projects and locations — check it before blaming your own deployment, and subscribe to alerts from it. Active Assist is the family of recommenders that analyze your actual usage and suggest optimizations: idle VM and idle disk recommendations, VM rightsizing, unattended-project cleanup, and IAM role recommendations that shrink over-broad grants. Cloud Hub is the newer Console home for operating applications as a whole, bringing application health signals and active events (incidents, maintenance) into one view.

Finally, Gemini Cloud Assist threads AI through operations: in Cloud Monitoring and Logging contexts you can ask it in natural language to help build queries, explain a spike on a chart, or run guided investigations that correlate an alert with related logs, errors, and recent changes to suggest probable root causes. The exam expects recognition, not mastery: symptom to tool. Latency across services — Trace; CPU or memory hot spots in code — Profiler; slow SQL — Query Insights; "is it Google or is it us" — Personalized Service Health; cost and IAM cleanup suggestions — Active Assist; AI-assisted investigation — Gemini Cloud Assist.