Observability in Cloud Native: Metrics, Logs, and Traces

In a traditional monolith running on one long-lived server, you could diagnose a problem by logging into the machine and reading a log file. Cloud native systems remove that option. A single user request may touch a dozen microservices, each running as containers in Pods that Kubernetes can restart, reschedule, or scale at any moment. The machine a container ran on five minutes ago may no longer exist. You cannot understand a system like this from the inside, so you have to understand it from the outside, through the telemetry it emits.

Observability is the property of a system that lets you infer its internal state from its external outputs. It is closely related to monitoring but not identical. Monitoring watches for failure modes you already anticipated: predefined dashboards, health checks, and thresholds. Observability goes further, giving you enough raw signal to ask new questions about problems you never predicted. For KCNA you need this distinction at recognition depth: monitoring answers known questions, observability lets you explore unknown ones.

The telemetry itself is conventionally grouped into three pillars: metrics, logs, and traces. Each pillar answers a different kind of question, and each has a well-known CNCF project behind it. A mature cloud native platform collects all three and correlates them, because no single pillar tells the whole story. The exam frequently asks you to match a scenario or a tool to the correct pillar, so keep the boundaries between them sharp as you read on.

Metrics are numeric measurements sampled over time: CPU usage, request rate, error count, queue depth. Because they are just numbers with labels, metrics are cheap to store at scale, easy to aggregate, and ideal for dashboards and alerting. Logs are timestamped records of discrete events, usually text or structured JSON: a request was served, an exception was thrown, a connection was refused. Logs carry rich detail about a single moment but are expensive to store and search in bulk. Traces record the journey of one request as it hops across services, broken into timed units called spans, and show you exactly where in a call chain the time went.

Kubernetes adds a fourth stream that is easy to confuse with logs: Events. An Event is an API object the control plane and kubelet record about changes to other objects, such as a Pod being scheduled, an image pull failing, a container being restarted or a probe failing. You read them with kubectl get events or at the bottom of kubectl describe. Events describe what Kubernetes did to a workload, not what the application printed, and they are short-lived: the API server keeps them for about an hour by default. When you need the error message or stack trace an application emitted, Events will not have it; that is in the container's logs.

A useful decision rule: reach for metrics when you want trends and alerts across many requests, reach for logs when you need the detail of a specific event, and reach for traces when a request crosses multiple services and you need to find the slow or failing hop. KCNA questions often hand you one of these needs and ask which pillar or tool fits.

Prometheus is the de facto metrics system of the cloud native world. It was the second project accepted into the CNCF, after Kubernetes itself, and it is a graduated project. Its defining design choice is the pull model: by default, instead of applications pushing measurements to a server, Prometheus periodically scrapes HTTP endpoints, conventionally a path named /metrics, that each target exposes in a simple text format. In Kubernetes, Prometheus discovers scrape targets automatically through service discovery, which suits an environment where Pods appear and disappear constantly. (Recent Prometheus versions can also be configured to accept pushed data, for example OTLP or remote-write, but scraping remains the default and the model the exam means.)

The one standard exception to pull is the Pushgateway. A short-lived batch job may finish before Prometheus ever scrapes it, so the job pushes its final metrics to the Pushgateway, which holds them, and Prometheus scrapes the Pushgateway like any other target. It is a cache for service-level batch jobs, not a general push path: routing long-running services through it would hide which instance is down and keep stale values after a job dies.

Scraped samples are stored in a local time-series database. Every series is identified by a metric name plus a set of key-value labels, such as the job, instance, or HTTP status code, and labels are what make slicing and aggregation possible. Local storage keeps history for a configurable retention period, 15 days by default; longer-term retention is usually handled by sending data to remote storage. You query this data with PromQL, the Prometheus query language. At KCNA depth you only need to recognize what PromQL is and what a query looks like, for example:

rate(http_requests_total{job="checkout"}[5m])

which returns the per-second request rate for the checkout job averaged over five minutes. A metric name followed by a label matcher in braces and a range in square brackets is the signature of PromQL. Several other observability tools have query languages that look similar, so know which belongs where:

Applications that do not natively expose Prometheus metrics are covered by exporters, sidecar or standalone processes that read a system's own statistics and re-expose them in the Prometheus format, so the system itself does not change. Well-known examples are node_exporter for host-level machine metrics and kube-state-metrics for the state of Kubernetes objects such as Deployments and Pods; many databases and message brokers have their own exporters.

Alerting is split into two parts. Prometheus itself evaluates alerting rules written in PromQL and fires alerts when a rule holds true. A rule can carry a for: duration, which only delays firing: the condition must stay true for that long before the alert moves from pending to firing, filtering out brief blips. It does nothing else. Firing alerts are sent to Alertmanager, a separate component that deduplicates them, groups related ones (for example, many per-instance alerts caused by one failing dependency become one notification), silences them during planned maintenance, and routes notifications to receivers such as email, Slack, or PagerDuty. Remember the division of labor: Prometheus decides that something is wrong and for how long it must persist; Alertmanager decides who hears about it, how, and how many times.

A classic KCNA trap is confusing Prometheus with metrics-server, because both deal in metrics but they serve entirely different purposes. The metrics-server is a lightweight cluster add-on that implements the Kubernetes Metrics API. It collects current CPU and memory usage for nodes and Pods from each node's kubelet and serves those numbers in memory, keeping no history at all. Its consumers are Kubernetes itself: the kubectl top command reads it to show you live resource usage, and the Horizontal Pod Autoscaler reads it to decide when to add or remove replicas based on resource utilization.

Because metrics-server keeps nothing but the latest snapshot, it cannot answer a question about last week or last month, and you cannot point PromQL at it. For resource history, Prometheus scrapes the container usage series that the kubelet exposes from its built-in cAdvisor component (CPU, memory and more per container) and stores them like any other series. Those series are what you graph or query when you need to compare today's usage with the past.

It also helps to know what does not depend on metrics-server. If it is removed, kubectl top stops working and the HPA can no longer scale on CPU or memory utilization. Everything else carries on: kubectl logs reads container log files through the kubelet, Prometheus scrapes its own targets, Grafana charts Prometheus data and Alertmanager routes alerts, none of them through the Metrics API.

The two are complementary, not competing. A production cluster typically runs both: metrics-server so that autoscaling and kubectl top work, and Prometheus so that engineers get dashboards, long-term trends, and alerts. If an exam question mentions kubectl top or resource-based autoscaling, the answer involves metrics-server; if it mentions scraping, PromQL, exporters, history or alert routing, the answer is the Prometheus stack.

The cloud native logging convention is simple: containers write their logs to stdout and stderr. The container runtime captures those streams and writes them to files on the node, and kubectl logs reads those files through the kubelet when you inspect a Pod. Writing to standard streams instead of custom log files matters because it decouples the application from log handling entirely; the app does not need to know where logs end up, and the platform can change the destination without touching application code. A log file the application writes inside its own container, in the writable layer, is invisible to kubectl logs and to node agents, and it disappears when the container is replaced.

Node-level logs alone are not enough, though. When a Pod is deleted or rescheduled, or a node is drained, the log files on that node stop being reachable through kubectl logs, and files are rotated away over time. The standard answer is a node-level logging agent deployed as a DaemonSet, so one agent runs on every node, tails the container log files, enriches each record with metadata such as the Pod and namespace, and ships everything to a centralized log store. That store, for example Elasticsearch, Grafana Loki, or a managed cloud logging service, is where engineers actually search and correlate logs across the whole cluster.

Workarounds such as a single central Deployment pulling logs through the API server, or a CronJob that runs kubectl logs, scale poorly, miss lines between runs and load the API server, which is why the DaemonSet pattern is the default.

The CNCF projects to know here are Fluentd, a graduated project, and Fluent Bit, its lightweight sibling written for low resource usage, which makes it a popular choice for the per-node agent role. Both collect, transform, and route log data between many sources and destinations. One more habit worth recognizing: emitting logs as structured JSON rather than free text makes them far easier to filter and query once centralized, and it is the norm in cloud native systems.

When a request enters a microservices system, it may pass through an API gateway, three or four services, a cache, and a database before a response leaves. If that request is slow, metrics tell you that latency is up and logs tell you what each service saw locally, but neither shows you the request's whole journey. Distributed tracing does. A trace is the record of one request end to end, composed of spans, where each span is a named, timed operation such as one service handling its part of the call. All spans of one request share a trace ID and point to their parent span, so they nest into a tree and the trace view reads like a timeline of exactly where the milliseconds went.

Making this work requires context propagation: each service must pass the trace ID and its current span ID along with outgoing calls, usually in HTTP headers (the W3C traceparent header is the common standard), so the next service's spans join the same trace. If a service does not forward that context, the next service starts a brand-new trace, and the backend shows fragments of one journey with no parent-child link between them.

That instrumentation work is where OpenTelemetry comes in. OpenTelemetry is the CNCF's vendor-neutral standard for generating and exporting telemetry, formed from the merger of the earlier OpenTracing and OpenCensus projects. It provides APIs and SDKs for many languages, which create spans and propagate context, plus the OpenTelemetry Collector, a pipeline component that receives, processes, and exports telemetry. The Collector is optional: SDKs can export straight to a backend over OTLP, and the Collector is added when you want central batching, filtering or fan-out to several backends. Because OpenTelemetry is vendor-neutral, you instrument your code once and can send the data to any compatible backend, avoiding lock-in. OpenTelemetry does not store or display traces itself.

Jaeger is the CNCF graduated project most associated with the backend side: it stores traces and provides the UI where you search for a request and inspect its span timeline. In a typical setup, applications instrumented with OpenTelemetry export spans, directly or through the Collector, into Jaeger. Grafana Tempo is another trace backend (queried with TraceQL). A log store such as Grafana Loki is not a trace backend: it can hold log lines that mention a trace ID, which helps correlate logs with traces, but it has no notion of spans, parent links or a request timeline. Prometheus is not one either; it stores numeric series, not spans.

Traces are voluminous, so systems commonly sample, keeping only a fraction of traces, which is an accepted trade-off at this level. Sampling decides whether a whole trace is kept or dropped; with the usual parent-based sampling, the decision travels in the propagated context so every service makes the same choice. Missing spans from sampling mean fewer traces, not one request split into several; a split trace points to broken context propagation.

Observability data is not only for firefighting; it is also how you manage performance and spend. On the performance side, a widely used starting point is the four golden signals: latency, traffic, errors, and saturation. Latency and error rates tell you what users experience, traffic tells you the load, and saturation tells you how close a resource is to its limit. All four are naturally expressed as Prometheus metrics, alerted on through Alertmanager, and visualized on dashboards, commonly with Grafana.

The signals work together when you diagnose. Latency is usually the symptom users notice; the other three help explain it. If latency rises while traffic and errors are flat, the load did not change and requests are not failing, so a resource bottleneck becomes a likely theory, and saturation is the signal that tests it: CPU throttling, memory pressure, or a full connection pool. To see whether saturation has changed, compare today's figures with a baseline, which needs history: the container usage series Prometheus has scraped from the kubelet's cAdvisor, queried with PromQL. kubectl top shows only the current moment.

Cost in Kubernetes is largely a story about resource requests. Every container can declare CPU and memory requests, and the scheduler reserves that capacity on a node whether or not the container uses it. Teams that guess high end up paying for idle headroom across the cluster; over-reserved capacity also blocks scheduling, so clusters scale out sooner than usage requires; teams that guess low risk throttling and evictions. The fix is empirical: compare requested resources against actual usage over time, using metrics, and right-size the requests. That same usage data feeds autoscaling, so accurate metrics directly translate into running only the capacity you need.

Be precise about what counts as evidence of idle reservation. Only requests compared with observed usage show capacity reserved but not used. Other comparisons are between configuration values, not usage: limits compared with requests describe how far a container may burst; a node's allocatable compared with its capacity is just the share reserved for the operating system, kubelet and eviction threshold (allocatable = capacity minus those reservations); and HPA target utilization compared with maximum replicas describes scaling policy. None of them shows whether a workload actually uses what it reserved.

For attributing spend, the CNCF hosts OpenCost, a project that measures the cost of Kubernetes workloads and allocates it by namespace, Deployment, label, or team, drawing on the same metrics pipeline. At KCNA depth you do not need pricing math; you need the concepts that overprovisioned requests waste money, that usage metrics are the evidence for right-sizing, and that cost monitoring tools exist in the cloud native ecosystem to make cluster spend visible.

Put the pillars together with a concrete incident. Your e-commerce platform runs a dozen microservices on Kubernetes. At 14:05, an alert fires: a Prometheus alerting rule on 95th percentile checkout latency crossed its threshold, and Alertmanager routed the page to your on-call channel. The metrics dashboard confirms the trend, latency climbing for ten minutes while traffic is normal, so this is not a load spike. Metrics have told you that something is wrong and roughly where to look, but not why.

Next you open a trace. In Jaeger you pull up recent slow checkout requests and read the span timeline. The gateway span is fast, the cart service span is fast, but the payment service span accounts for nearly all of the elapsed time on every slow trace. Tracing has localized the problem to one hop in the chain, something neither metrics nor logs could show you across service boundaries.

Finally you go to the logs. Searching the centralized store, shipped there by the Fluent Bit DaemonSet, filtered to the payment service Pods for the last fifteen minutes, you find repeated connection pool exhaustion errors against the payment provider. A PromQL query over the payment Pods' container CPU and memory series in Prometheus shows usage well within its normal historical range, ruling out resource saturation; kubectl top would only have shown the current moment. The fix is a configuration change, and after rollout the latency metric returns to baseline and the alert resolves. Each pillar answered its own question: metrics detected and quantified, traces localized, logs explained.