What is Apache Kafka?
Apache Kafka is an open-source distributed event-streaming platform for publishing, storing and processing streams of events — records of something that happened — in real time and at scale. It is built around a durable, ordered log: unlike a typical message queue, events written to Kafka stay there for a configured period rather than disappearing the moment a consumer reads them, which lets multiple independent systems read the same stream, replay history, and process events at their own pace. This guide covers the publish/subscribe log model at a beginner level, the situations Kafka gets used for, the managed options that avoid running it yourself, and how it differs from a plain message queue.
The pub/sub-plus-log model
Kafka organises events into topics — named streams of related events, such as "orders placed" or "page views." Producers are the systems that write events onto a topic, and consumers are the systems that read from it; any number of consumers can read the same topic independently, each keeping track of their own position in the stream. What makes Kafka distinct from a simple publish/subscribe system is that it stores events as an ordered, append-only log for a retention period you configure, rather than deleting them once delivered — a new consumer can start reading from the beginning of that retained history, and an existing one can be restarted without losing its place.
Why "distributed" matters here
Kafka runs across a cluster of machines, with each topic split into partitions spread across that cluster — this is what lets it handle very large volumes of events with high throughput, and keep working even if an individual machine in the cluster fails. Our what is big data explainer covers the broader idea of distributing storage and processing across many machines, which Kafka applies specifically to continuous streams of events rather than static datasets.
Common uses
A few patterns account for most real-world Kafka deployments:
- Event streaming — capturing a continuous flow of events (orders, clicks, sensor readings) as they happen, for any number of downstream systems to consume.
- Real-time data pipelines — feeding events into analytics, monitoring or machine learning systems with low delay, rather than waiting for a scheduled batch job.
- Decoupling systems — letting services publish events without knowing or caring which other services will consume them, so producers and consumers can be added, removed or changed independently.
Kafka versus a plain message queue
This is the distinction people ask about most, and it is worth stating precisely: a traditional message queue typically removes a message once a consumer has processed it, and is generally built around one message going to one consumer (or one consumer group). Kafka retains events in an ordered log for a set retention window regardless of whether they have been read, and happily supports many independent consumers reading the same events at different times. That makes Kafka a better fit when multiple systems need the same stream of events, or when replaying history matters; a plain queue is often simpler and is a natural fit for straightforward task distribution, where each message should be handled once and then be done with. Our what is a message queue explainer covers that simpler model in more depth, and our what is a data pipeline explainer covers how event streams like Kafka’s feed into broader pipelines.
Managed Kafka, so you don’t run it yourself
Running a Kafka cluster directly means managing distributed infrastructure — brokers, partitions, replication — which is real operational work. Managed options remove most of that: Amazon MSK (Managed Streaming for Apache Kafka) runs Kafka as a managed AWS service, Confluent Cloud offers a fully managed Kafka service founded by Kafka’s original creators, and Azure Event Hubs offers a Kafka-compatible endpoint, letting Kafka client applications talk to Event Hubs without code changes even though the underlying service is not Kafka itself. Each option lets teams use the Kafka model without operating the cluster by hand.
Where this appears in cert study
Event streaming and its trade-offs against traditional queues are core material in the AWS Data Engineer Associate exam, which covers Amazon MSK and Kinesis alongside batch-processing services. Microsoft’s Fabric Data Engineer Associate exam covers the equivalent streaming concepts on Azure, including Event Hubs’ Kafka-compatible endpoint. The Kubernetes and Cloud Native Associate (KCNA) exam does not test Kafka directly, but candidates frequently encounter Kafka running as a containerised workload on Kubernetes in real deployments, which is why it is a useful concept to have alongside container orchestration knowledge.
Original practice questions, timed mock exams and revision notes. No card, nothing to pay.