Data Ingestion on AWS: Kinesis, MSK, DMS, Glue, and Batch Patterns

The streaming sources DEA-C01 tests are Amazon Kinesis Data Streams, Amazon MSK (Managed Streaming for Apache Kafka), DynamoDB Streams, and AWS DMS running change data capture (CDC) — each fits a different producer and retention profile.

Kinesis Data Streams is the AWS-native default. In provisioned mode, each shard accepts 1 MB/s or 1,000 records per second of writes and serves 2 MB/s of reads shared across all standard consumers. Retention defaults to 24 hours and can be extended up to 365 days, which is what makes Kinesis pipelines replayable. On-demand capacity mode removes shard management and scales automatically — the answer when the question says traffic is unpredictable and you want the least operational overhead.

Amazon MSK is the choice when the scenario mentions Apache Kafka: existing Kafka producers or consumers, the Kafka ecosystem (Kafka Connect, exactly-once Kafka semantics), or migrating a self-managed Kafka cluster. MSK Serverless removes broker sizing. If nothing in the question requires Kafka compatibility, Kinesis is usually the simpler, lower-overhead answer.

DynamoDB Streams emits an ordered, per-item record of table changes with a fixed 24-hour retention, consumed almost always by Lambda. If a scenario needs longer retention or many consumers of table changes, the alternative is Kinesis Data Streams for DynamoDB, which forwards changes into a regular Kinesis stream.

AWS DMS reads a relational database's transaction log and streams inserts, updates, and deletes continuously (CDC) to targets including S3, Kinesis, and Redshift — the standard answer for keeping an analytics store in sync with an operational database without repeated full extracts. AWS Glue streaming jobs (Spark Structured Streaming) can also consume Kinesis or Kafka directly when the transformation itself must run continuously.

Batch ingestion on AWS reads bounded datasets on a schedule, and the services to know are Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, and Amazon AppFlow.

S3 is the landing zone for nearly every batch pipeline: files arrive from uploads, DMS full loads, or partner transfers, and downstream engines read them in place. Glue jobs are the workhorse readers, and their key batch configuration is the job bookmark: with bookmarks enabled, each run processes only data added since the last successful run, turning a naive full re-read into an incremental load. Glue crawlers keep the Data Catalog schema and partitions current as new files land.

Redshift COPY is the correct way to bulk-load the warehouse — never row-by-row INSERT statements. COPY loads in parallel across slices, so split input into multiple files of roughly equal size (ideally a multiple of the slice count, compressed, around 100 MB–1 GB each) rather than one giant file. A manifest file makes the load list explicit and repeatable. Redshift auto-copy can watch an S3 path and load new files automatically.

DMS full load performs the initial bulk migration of a database, usually followed by CDC for ongoing changes. Amazon AppFlow is the no-code answer for SaaS sources — Salesforce, ServiceNow, Google Analytics — flowing into S3 or Redshift on a schedule or on demand, with field mapping and filtering built in. Lambda suits small, event-driven batch pulls, such as calling a REST data API on a schedule and writing the response to S3; remember its 15-minute execution ceiling. EMR reads batch data at scale when you need full Spark or Hadoop control.

Across all of these, the recurring configuration decisions are format (prefer columnar), compression, partitioning of the landing path, and incremental markers (bookmarks, manifests, CDC checkpoints) so a run never double-processes data.

Choose streaming when the value of data decays in seconds to minutes; choose batch when hourly or daily freshness is acceptable and you want lower cost and simpler operations. DEA-C01 signals the intended answer with latency words — read them carefully.

Two gotchas. First, Firehose is delivery, not a message bus: it buffers by size or time interval and writes to a destination, but it has no consumer API, no ordering guarantee you can consume against, and no replay — if multiple applications must read the same events, you need Kinesis Data Streams or MSK. Second, micro-batching is still batch: a Glue job running every five minutes may satisfy a "near-real-time" requirement more cheaply than a always-on streaming job, and cost-focused questions reward noticing that.

You start batch jobs and crawlers either on a schedule — Amazon EventBridge or Apache Airflow — or in response to an event, and the exam expects you to pick the trigger that matches the workflow's complexity.

EventBridge Scheduler runs cron and rate expressions against almost any AWS API target: start a Glue job or crawler nightly, invoke Lambda every five minutes, launch a Step Functions state machine. It is serverless and the least-overhead answer for simple time-based orchestration. Glue also has native triggers: scheduled (cron), on-demand, conditional (start job B when jobs A and C both succeed), and EventBridge-event triggers — conditional triggers chain jobs into a Glue workflow without external orchestration.

Amazon MWAA (Managed Workflows for Apache Airflow) is the answer when the scenario describes complex dependencies, dozens of interrelated tasks, backfills, or a team already writing Airflow DAGs in Python. Airflow gives you rich dependency graphs and retry semantics, but it runs an always-on environment — more capability, more cost and operational surface than EventBridge.

Event triggers make pipelines reactive instead of scheduled. S3 Event Notifications fire on object creation or deletion and deliver directly to Lambda, SQS, or SNS — the classic pattern is file lands in S3, Lambda starts a Glue job. Routing S3 events through EventBridge (enabled per bucket) adds richer filtering on object attributes, fan-out to many targets, archive and replay of events, and direct integration with targets such as Step Functions. If the question needs multiple consumers of the same bucket event or content-based rules, EventBridge is the better answer; for a single simple destination, plain S3 notifications are cheaper and simpler.

Prefer event triggers over polling schedules when files arrive unpredictably — you eliminate both the latency of waiting for the next scheduled run and the wasted runs that find nothing to process.

Lambda consumes a Kinesis stream through an event source mapping (ESM): Lambda polls the shards for you and invokes your function synchronously with batches of records, and tuning that mapping is a favorite DEA-C01 topic.

The knobs that matter: batch size (records per invocation) and batching window (how long to wait to fill a batch) trade latency against per-invocation efficiency. By default one batch per shard is processed at a time, so a 10-shard stream yields at most 10 concurrent invocations; raising the parallelization factor (up to 10 concurrent batches per shard) multiplies throughput while still preserving order per partition key. The starting position (TRIM_HORIZON, LATEST, or a timestamp) controls where a new consumer begins reading.

Error handling is where candidates lose marks. A failed batch is retried and blocks that shard until it succeeds or expires — one bad record can stall a shard for the full retention period. The fixes: bisect batch on function error (split the failing batch to isolate the poison record), maximum retry attempts and maximum record age to bound retries, an on-failure destination (SQS or SNS) to capture failed records for later analysis, and ReportBatchItemFailures so a partially successful batch only reprocesses from the first failed record.

Scenario

A payments team streams transaction events through a 4-shard Kinesis stream into Lambda for enrichment. Processing lags during peaks, and one malformed record recently froze a shard for hours. The exam-correct fix: raise the parallelization factor to increase concurrency without resharding, enable bisect-on-error with a bounded retry count, and route poison records to an SQS on-failure destination — throughput recovers and a bad record can no longer block the pipeline.

Throttling means the source is rejecting requests faster than its provisioned capacity, and the exam tests service-specific remedies for DynamoDB, Kinesis, and Amazon RDS.

DynamoDB throws ProvisionedThroughputExceededException when reads or writes exceed capacity — often caused by a hot partition key, where traffic concentrates on few keys. Remedies in order of exam preference: retry with exponential backoff and jitter (the AWS SDKs do this automatically), switch to on-demand capacity for spiky workloads, redesign the partition key for even distribution (add a random or calculated suffix), and cache hot reads with DAX. Adding more total capacity does not fix a hot partition — that distinction is tested.

Kinesis write throttling means producers exceed a shard's 1 MB/s / 1,000 records-per-second limit: split shards (or switch to on-demand mode), and aggregate small records with the Kinesis Producer Library so many logical records fit one API record. Read throttling — multiple consumers competing for the shared 2 MB/s per shard — is solved with enhanced fan-out, covered next.

RDS rate limits usually surface as connection exhaustion, classically from hundreds of Lambda invocations each opening their own connection. The answer is Amazon RDS Proxy, which pools and shares connections; read replicas offload read-heavy ingestion queries. Whatever the source, batch your ingestion reads instead of row-at-a-time calls.

IP allowlists control who may connect to a data source. Inside a VPC that means security groups (and network ACLs); Glue reaches VPC databases through a connection that attaches ENIs inside your subnets, so allow the ENIs' security group on the database. To call an external source that allowlists caller IPs, route egress through a NAT gateway so all traffic presents the NAT's stable Elastic IP — that single address is what the vendor allowlists.

Fan-in is many producers writing into one stream; fan-out is one stream feeding many consumers — and managing both without throttling is an explicit DEA-C01 skill.

Fan-in concerns are write capacity and record efficiency. Thousands of devices or microservices writing tiny records individually burn the per-shard records-per-second limit long before the megabyte limit; the fix is aggregation — the Kinesis Producer Library packs many logical records into each PutRecords payload — plus a partition key that spreads writes evenly across shards so no shard becomes hot while others idle.

Fan-out on Kinesis has two modes. Standard consumers share each shard's 2 MB/s read throughput and poll over HTTP — with three or more consumers they contend and fall behind. Enhanced fan-out gives each registered consumer its own dedicated 2 MB/s per shard, pushed over HTTP/2 with typically ~70 ms latency, at additional cost. The exam pattern: if a scenario adds a second or third real-time consumer and mentions rising read latency or ReadProvisionedThroughputExceeded, enhanced fan-out is the answer — not adding shards, which only raises write capacity.

For message-based (non-stream) distribution, the SNS-to-SQS fan-out pattern delivers one published event to many independent SQS queues, each consumed at its own pace with its own retry behavior. On MSK, fan-out is native to Kafka: each consumer group receives the full topic independently.

Fan-out is also an architectural decoupler: a common pattern streams events once into Kinesis, with one consumer delivering raw data to S3 via Firehose (the archive), another powering real-time alerting through Lambda, and a third feeding Managed Service for Apache Flink analytics — each consumer isolated from the others' failures.

A replayable ingestion pipeline can reprocess past data after a bug or outage, and replayability comes from sources that retain records after delivery. Kinesis Data Streams retains records for its configured retention (24 hours up to 365 days) regardless of consumption, so a consumer can rewind to a sequence number or timestamp and re-read. Kafka on Amazon MSK is equally replayable through per-topic retention and consumer offsets. By contrast, SQS deletes a message once a consumer processes it, and Firehose delivers and moves on — neither can replay. When the exam asks which design lets you reprocess events after fixing a defective transformation, the answer is a retained stream (Kinesis or MSK), often paired with a durable raw copy in S3 — landing immutable raw data in S3 is the ultimate replay mechanism, since any batch engine can reprocess it at any time. Replay makes duplicates inevitable, so downstream processing must be idempotent: reprocessing the same record twice must not corrupt results.

Stateless vs stateful transactions divide processing by whether context is needed. A stateless operation handles each record independently — validate, mask a field, convert format, route — so it scales horizontally without coordination and suits Lambda perfectly. A stateful operation needs memory across records: windowed aggregations (orders per minute), deduplication, sessionization, and stream joins. Stateful streaming needs somewhere durable to keep that state — Amazon Managed Service for Apache Flink checkpoints operator state for exactly-once processing, Spark Structured Streaming (Glue streaming jobs) checkpoints to S3, and Lambda offers lightweight tumbling-window aggregation per shard. If a question involves windows, sessions, or joins across events, plain stateless Lambda is the wrong answer; reach for Flink or Spark.