Observability in Cloud Native: Metrics, Logs, and Traces

In a traditional monolith running on one long-lived server, you could diagnose a problem by logging into the machine and reading a log file. Cloud native systems remove that option. A single user request may touch a dozen microservices, each running as containers in Pods that Kubernetes can restart, reschedule, or scale at any moment. The machine a container ran on five minutes ago may no longer exist. You cannot understand a system like this from the inside, so you have to understand it from the outside, through the telemetry it emits.

Observability is the property of a system that lets you infer its internal state from its external outputs. It is closely related to monitoring but not identical. Monitoring watches for failure modes you already anticipated: predefined dashboards, health checks, and thresholds. Observability goes further, giving you enough raw signal to ask new questions about problems you never predicted. For KCNA you need this distinction at recognition depth: monitoring answers known questions, observability lets you explore unknown ones.

The telemetry itself is conventionally grouped into three pillars: metrics, logs, and traces. Each pillar answers a different kind of question, and each has a well-known CNCF project behind it. A mature cloud native platform collects all three and correlates them, because no single pillar tells the whole story. The exam frequently asks you to match a scenario or a tool to the correct pillar, so keep the boundaries between them sharp as you read on.

Metrics are numeric measurements sampled over time: CPU usage, request rate, error count, queue depth. Because they are just numbers with labels, metrics are cheap to store at scale, easy to aggregate, and ideal for dashboards and alerting. Logs are timestamped records of discrete events, usually text or structured JSON: a request was served, an exception was thrown, a connection was refused. Logs carry rich detail about a single moment but are expensive to store and search in bulk. Traces record the journey of one request as it hops across services, broken into timed units called spans, and show you exactly where in a call chain the time went.

A useful decision rule: reach for metrics when you want trends and alerts across many requests, reach for logs when you need the detail of a specific event, and reach for traces when a request crosses multiple services and you need to find the slow or failing hop. KCNA questions often hand you one of these needs and ask which pillar or tool fits.

Prometheus is the de facto metrics system of the cloud native world. It was the second project accepted into the CNCF, after Kubernetes itself, and it is a graduated project. Its defining design choice is the pull model: instead of applications pushing measurements to a server, Prometheus periodically scrapes HTTP endpoints, conventionally a path named /metrics, that each target exposes in a simple text format. In Kubernetes, Prometheus discovers scrape targets automatically through service discovery, which suits an environment where Pods appear and disappear constantly.

Scraped samples are stored in a time-series database. Every series is identified by a metric name plus a set of key-value labels, such as the job, instance, or HTTP status code, and labels are what make slicing and aggregation possible. You query this data with PromQL, the Prometheus query language. At KCNA depth you only need to recognize what PromQL is and what a query looks like, for example:

rate(http_requests_total{job="checkout"}[5m])

which returns the per-second request rate for the checkout job averaged over five minutes. Applications that do not natively expose Prometheus metrics are covered by exporters, sidecar or standalone processes that translate third-party systems into the Prometheus format. Well-known examples are node_exporter for host-level machine metrics and kube-state-metrics for the state of Kubernetes objects such as Deployments and Pods.

Alerting is split into two parts. Prometheus itself evaluates alerting rules written in PromQL and fires alerts when a rule holds true. Those alerts are sent to Alertmanager, a separate component that deduplicates, groups, and silences them, then routes notifications to receivers such as email, Slack, or PagerDuty. Remember the division of labor: Prometheus decides that something is wrong, Alertmanager decides who hears about it and how.

A classic KCNA trap is confusing Prometheus with metrics-server, because both deal in metrics but they serve entirely different purposes. The metrics-server is a lightweight cluster add-on that implements the Kubernetes Metrics API. It collects current CPU and memory usage for nodes and Pods from each node's kubelet and serves those numbers in memory, keeping no history at all. Its consumers are Kubernetes itself: the kubectl top command reads it to show you live resource usage, and the Horizontal Pod Autoscaler reads it to decide when to add or remove replicas based on resource utilization.

The two are complementary, not competing. A production cluster typically runs both: metrics-server so that autoscaling and kubectl top work, and Prometheus so that engineers get dashboards, long-term trends, and alerts. If an exam question mentions kubectl top or resource-based autoscaling, the answer involves metrics-server; if it mentions scraping, PromQL, exporters, or alert routing, the answer is the Prometheus stack.

The cloud native logging convention is simple: containers write their logs to stdout and stderr, and nothing else. The container runtime captures those streams and writes them to files on the node, which is what kubectl logs reads when you inspect a Pod. Writing to standard streams instead of custom log files matters because it decouples the application from log handling entirely; the app does not need to know where logs end up, and the platform can change the destination without touching application code.

Node-level logs alone are not enough, though. When a Pod is deleted or rescheduled, or a node is drained, the log files on that node stop being reachable through kubectl logs, and files are rotated away over time. The standard answer is a node-level logging agent deployed as a DaemonSet, so one agent runs on every node, tails the container log files, enriches each record with metadata such as the Pod and namespace, and ships everything to a centralized log store. That store, for example Elasticsearch, Grafana Loki, or a managed cloud logging service, is where engineers actually search and correlate logs across the whole cluster.

The CNCF projects to know here are Fluentd, a graduated project, and Fluent Bit, its lightweight sibling written for low resource usage, which makes it a popular choice for the per-node agent role. Both collect, transform, and route log data between many sources and destinations. One more habit worth recognizing: emitting logs as structured JSON rather than free text makes them far easier to filter and query once centralized, and it is the norm in cloud native systems.

When a request enters a microservices system, it may pass through an API gateway, three or four services, a cache, and a database before a response leaves. If that request is slow, metrics tell you that latency is up and logs tell you what each service saw locally, but neither shows you the request's whole journey. Distributed tracing does. A trace is the record of one request end to end, composed of spans, where each span is a named, timed operation such as one service handling its part of the call. Spans nest into a tree, so the trace view reads like a timeline of exactly where the milliseconds went.

Making this work requires context propagation: each service must pass trace identifiers along with outgoing calls, usually in HTTP headers, so the backend can stitch spans from different services into one trace. That is instrumentation work, and it is where OpenTelemetry comes in. OpenTelemetry is the CNCF's vendor-neutral standard for generating and exporting telemetry, formed from the merger of the earlier OpenTracing and OpenCensus projects. It provides APIs and SDKs for many languages plus the OpenTelemetry Collector, a pipeline component that receives, processes, and exports telemetry. Because it is vendor-neutral, you instrument your code once and can send the data to any compatible backend, avoiding lock-in.

Jaeger is the CNCF graduated project most associated with the backend side: it stores traces and provides the UI where you search for a request and inspect its span timeline. In a typical setup, applications instrumented with OpenTelemetry export spans, optionally through the Collector, into Jaeger for storage and visualization. Traces are voluminous, so systems commonly sample, keeping only a fraction of traces, which is an accepted trade-off at this level.

Observability data is not only for firefighting; it is also how you manage performance and spend. On the performance side, a widely used starting point is the four golden signals: latency, traffic, errors, and saturation. Latency and error rates tell you what users experience, traffic tells you the load, and saturation tells you how close a resource is to its limit. All four are naturally expressed as Prometheus metrics, alerted on through Alertmanager, and visualized on dashboards, commonly with Grafana.

Cost in Kubernetes is largely a story about resource requests. Every container can declare CPU and memory requests, and the scheduler reserves that capacity whether or not the container uses it. Teams that guess high end up paying for idle headroom across the cluster; teams that guess low risk throttling and evictions. The fix is empirical: compare requested resources against actual usage over time, using metrics, and right-size the requests. That same usage data feeds autoscaling, so accurate metrics directly translate into running only the capacity you need.

For attributing spend, the CNCF hosts OpenCost, a project that measures the cost of Kubernetes workloads and allocates it by namespace, Deployment, label, or team, drawing on the same metrics pipeline. At KCNA depth you do not need pricing math; you need the concepts that overprovisioned requests waste money, that usage metrics are the evidence for right-sizing, and that cost monitoring tools exist in the cloud native ecosystem to make cluster spend visible.

Put the pillars together with a concrete incident. Your e-commerce platform runs a dozen microservices on Kubernetes. At 14:05, an alert fires: a Prometheus alerting rule on 95th percentile checkout latency crossed its threshold, and Alertmanager routed the page to your on-call channel. The metrics dashboard confirms the trend, latency climbing for ten minutes while traffic is normal, so this is not a load spike. Metrics have told you that something is wrong and roughly where to look, but not why.

Next you open a trace. In Jaeger you pull up recent slow checkout requests and read the span timeline. The gateway span is fast, the cart service span is fast, but the payment service span accounts for nearly all of the elapsed time on every slow trace. Tracing has localized the problem to one hop in the chain, something neither metrics nor logs could show you across service boundaries.

Finally you go to the logs. Searching the centralized store, shipped there by the Fluent Bit DaemonSet, filtered to the payment service Pods for the last fifteen minutes, you find repeated connection pool exhaustion errors against the payment provider. A quick kubectl top check via metrics-server shows the Pods are not resource-starved, ruling out CPU saturation. The fix is a configuration change, and after rollout the latency metric returns to baseline and the alert resolves. Each pillar answered its own question: metrics detected and quantified, traces localized, logs explained.