A retail company captures website clickstream events. Three applications must each process every event within seconds, reading at their own pace. A fraud-model team plans to add a fourth application next month, and it must be able to start from the events captured earlier that same day. Which ingestion service should the data engineer use?
Choose one.
Kinesis Data Streams is a retained, multi-consumer stream; Firehose is delivery only, SQS is a competing-consumer queue, and SNS standard topics deliver only to current subscribers.
Two requirements decide it: every application reads every event independently, and a consumer added later must read events from earlier in the day. A Kinesis data stream keeps records for its retention period regardless of consumption, so any number of consumers read the same data at their own positions and a new one can start from the oldest retained record. SNS plus SQS meets the first requirement but not the second, a shared SQS queue meets neither, and Firehose exposes no read API at all.
- Note the latency (seconds) and that every application needs every event.
- Note the late-joining consumer: it needs events that were published before it existed.
- Eliminate designs without retention a new reader can rewind into (SNS, SQS, Firehose).
- Choose the retained multi-consumer stream: Kinesis Data Streams.
Exam tip: Many independent readers plus a late joiner that needs past events means a retained stream, not a queue, topic or delivery stream.
Data Ingestion on AWS: Kinesis, MSK, DMS, Glue, and Batch Patterns — the lesson that teaches this.