Data Ingestion on AWS: Kinesis, MSK, DMS, Glue, and Batch Patterns

The streaming sources DEA-C01 tests are Amazon Kinesis Data Streams, Amazon MSK (Managed Streaming for Apache Kafka), DynamoDB Streams, and AWS DMS running change data capture (CDC) — each fits a different producer and retention profile — with Amazon Data Firehose and SQS/SNS as the delivery and messaging alternatives you must tell them apart from.

Kinesis Data Streams is the AWS-native default. In provisioned mode, each shard accepts 1 MB/s or 1,000 records per second of writes and serves 2 MB/s of reads shared across all standard consumers. Retention defaults to 24 hours and can be extended up to 365 days, which is what makes Kinesis pipelines replayable. On-demand capacity mode removes shard management and scales automatically, which suits unpredictable traffic when you want the least operational overhead. It instantly absorbs up to double the stream's previous peak write throughput; a sudden jump far beyond that can still throttle briefly while it scales.

Amazon MSK is the choice when the scenario needs Apache Kafka: existing Kafka producers or consumers, the Kafka ecosystem (Kafka Connect, Kafka transactions), or migrating a self-managed Kafka cluster. MSK Serverless removes broker sizing. If nothing requires Kafka compatibility, Kinesis is usually the simpler, lower-overhead choice.

DynamoDB Streams emits an ordered, per-item record of table changes with a fixed 24-hour retention, consumed almost always by Lambda. The stream view type decides what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES — only the last lets a consumer compare before and after values (for example, to emit just the attributes that changed). For longer retention or many consumers, use Kinesis Data Streams for DynamoDB, which forwards item changes into a regular Kinesis stream. DMS and DynamoDB change capture are covered in more depth in the next section.

AWS Glue streaming jobs (Spark Structured Streaming) read from Kinesis Data Streams and from Apache Kafka (including MSK) — not from DynamoDB Streams directly. To process DynamoDB changes in a Glue streaming job, route them through Kinesis Data Streams for DynamoDB first. Amazon Redshift can also read streams itself: streaming ingestion maps a Kinesis stream or MSK topic into a materialized view, with no staging in S3.

Streams versus queues

A partitioned stream and a work queue scale consumers in different ways, and the difference decides many design questions.

Choose a queue when each item is an independent unit of work that should be retried on its own and order does not matter; choose a stream when several applications must read the same ordered events or you need replay.

Change data capture (CDC) reads a database's own change log so inserts, updates and deletes reach the analytics store without repeated full extracts; on AWS that means AWS DMS for relational sources and DynamoDB Streams for DynamoDB tables.

AWS DMS has three task types, and the scenario's wording picks one:

• Full load — copies the existing data once, then stops. A one-time migration or snapshot.
• CDC only — replicates ongoing changes from a start point, for when the initial copy was made another way (for example, a native backup restore).
• Full load and ongoing replication — copies the existing data, then keeps applying changes. The usual answer for "migrate and keep in sync".

Source prerequisites matter. DMS reads MySQL changes from the binary log, so binary logging must be on and binlog_format must be ROW (with a full row image). On Amazon RDS for MySQL, binary logging is turned on only when automated backups are enabled (a backup retention period above zero), and the binlog format is set in a custom DB parameter group. A CDC task against a source without these settings fails or captures nothing.

With an S3 target, DMS writes changes, not a current copy. Each CDC record lands in a file with an Op column — I, U or D — plus the row values. S3 objects cannot be updated in place, so something downstream must apply those records: typically a Glue or EMR job that runs a MERGE into an Apache Iceberg (or other open table format) table, producing a queryable current state.

Why not just poll the table? Key-based incremental reads — a Glue JDBC job bookmark on an auto-incrementing key, or a "max id" watermark — only see new rows. They miss updates to rows already read and never see deletes. A last-modified timestamp column catches updates but still misses hard deletes. Only log-based CDC captures all three operations.

DynamoDB change capture is DynamoDB Streams (24-hour retention, typically consumed by Lambda) or Kinesis Data Streams for DynamoDB (longer retention, more consumers, usable by Glue streaming and Firehose). Pick the stream view type that carries what the consumer needs — NEW_AND_OLD_IMAGES when it must compare old and new values.

Batch ingestion on AWS reads bounded datasets on a schedule or when files arrive, and the services to know are Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, and Amazon AppFlow; the configuration choices that matter are incremental markers, file layout for loading, and concurrency.

S3 is the landing zone for nearly every batch pipeline: files arrive from uploads, DMS full loads, Firehose delivery, or partner transfers, and downstream engines read them in place. Glue crawlers keep the Data Catalog schema and partitions current as new files land. EMR reads batch data at scale when you need full Spark or Hadoop control. Lambda suits small, event-driven pulls — such as calling a REST API on a schedule and writing the response to S3 — within its 15-minute execution limit.

Glue job bookmarks and job run queuing

A job bookmark records what a Glue job has already processed. For S3 sources it tracks the objects read, so each run reads only new files instead of the whole prefix; for JDBC sources it tracks the highest value of bookmark keys (by default the primary key), which must increase monotonically — so, as above, it catches new rows but not updates or deletes. Bookmarks advance only when the job commits at the end of a successful run.

Job run queuing is unrelated to what data is read: when it is on, a run that would exceed the account's or the job's concurrency or capacity limits waits in a queue instead of failing. Use bookmarks to stop reprocessing input; use queuing to stop runs failing at concurrency limits.

Loading Amazon Redshift

Bulk loads into Redshift use COPY from S3 (or DynamoDB, EMR, or remote hosts over SSH) — not INSERT statements, which run row by row through the leader node. UNLOAD is the opposite direction: it exports query results to S3. COPY loads in parallel across the cluster's slices, so input layout decides speed:

• A large uncompressed delimited file is split into scan ranges automatically, so several slices share it. Columnar Parquet and ORC files of 128 MB or more are split the same way.
• A compressed file cannot be split — one slice reads it all. Split compressed input into multiple files of roughly equal size, ideally a multiple of the slice count (AWS suggests about 1 MB–1 GB each after compression).
• A manifest lists exactly which files to load, which makes loads explicit and repeatable when a prefix holds extra objects. Mark an entry "mandatory": true and COPY fails if that file is missing, instead of silently loading less. A manifest does not make one big compressed file load in parallel.

Redshift auto-copy (a COPY JOB) can watch an S3 path and load new files automatically.

Shaping data on the way in with Firehose

Amazon Data Firehose can do batch-friendly work during delivery: record format conversion turns JSON into Apache Parquet or ORC using a schema from a Glue Data Catalog table, and dynamic partitioning writes records to S3 prefixes built from fields inside the records (for example, customer_id=…/date=…/). Both remove the need for a separate job just to convert or partition landed files.

Across all of these, the recurring decisions are format (prefer columnar), compression, partitioning of the landing path, and incremental markers (bookmarks, manifests, CDC checkpoints, AppFlow incremental transfer) so a run never double-processes data.

Amazon AppFlow is a managed, no-code integration service that moves data between SaaS applications — Salesforce, ServiceNow, Zendesk, Google Analytics and others — and AWS destinations such as S3 and Redshift, with field mapping, filtering and validation built in; it replaces custom API code for supported connectors.

A flow has one of three triggers:

Incremental transfer is available only on scheduled flows, and it filters on a timestamp field you choose. The field decides what you catch: CreatedDate picks up only new records, so edits to existing ones never arrive; LastModifiedDate (or another last-updated field) catches both new and modified records.

Event-triggered Salesforce flows depend on Salesforce Change Data Capture: the object must be selected for Change Data Capture in Salesforce, or the flow activates but never runs. AppFlow also works in the other direction — Salesforce is a supported destination, so a flow can push records from S3 into Salesforce without custom code.

Consuming a data API means calling a provider's HTTP endpoint repeatedly and reliably — following pagination, staying under its rate limit, retrying throttled calls, and resuming after a failure without losing or duplicating data.

• Pagination. APIs return large results a page at a time with a next-page token, cursor or offset. Keep requesting until no token comes back; a cursor is safer than an offset when data changes during the read.
• Pace yourself. If the provider allows N requests per second per key, every worker in the job shares that one budget, so put a client-side rate limiter (for example, a token bucket) in front of all of them and keep the combined rate below N. Adding threads or concurrent invocations without a shared limit just produces more throttling.
• Handle HTTP 429. "Too Many Requests" (and 503) are retryable: honour a Retry-After header when present, otherwise retry with exponential backoff with jitter — growing, randomised waits so many clients do not retry in lockstep — and cap the number of attempts so a persistent failure surfaces instead of looping forever.
• Make runs resumable. Persist a checkpoint — the last successful page cursor — in a durable store such as DynamoDB or S3 after each page, so a failed run restarts where it stopped instead of from page one.
• Write idempotently. Key each page's output deterministically (for example, an S3 object named by run and page number) so re-fetching a page overwrites its earlier output instead of creating a duplicate.

Where it fits, a managed connector beats custom code: AppFlow for supported SaaS sources, and Lambda or a Glue Python shell job on a schedule for the rest.

Choose streaming when the value of data decays in seconds to minutes; choose batch when hourly or daily freshness is acceptable and you want lower cost and simpler operations. DEA-C01 signals the intended model with latency words — read them carefully.

Two gotchas. First, Firehose is delivery, not a message bus: it buffers by size or time interval and writes to a destination, but it has no consumer API and no replay — if multiple applications must read the same events, you need Kinesis Data Streams or MSK. Second, micro-batching is still batch: a Glue job running every few minutes may satisfy a "near-real-time" requirement more cheaply than an always-on streaming job, and cost-focused questions reward noticing that.

You start batch jobs and crawlers either on a schedule — Amazon EventBridge, Glue's own triggers, or Apache Airflow — or in response to an event, and the right choice depends on time-zone needs, dependencies between steps, and how many targets react to the event.

Time-based schedules

A UTC cron runs at a fixed UTC time, so the local time drifts by an hour when daylight saving changes. If a job must run at a local wall-clock time all year, use EventBridge Scheduler with a time zone.

Chaining crawlers and jobs

Glue triggers come in four kinds: scheduled, on-demand, conditional, and EventBridge-event. A Glue workflow chains them: a conditional trigger can watch a crawler and start a job only when the crawl state is SUCCEEDED (or start job B when jobs A and C both succeed), with no code. The heavier alternative is an EventBridge rule on the Glue "Crawler State Change" event that invokes a Lambda function to start the job — it works, but it is more to build and maintain than a workflow.

Amazon MWAA (Managed Workflows for Apache Airflow) fits when the scenario describes complex dependencies, dozens of interrelated tasks, backfills, or a team already writing Airflow DAGs. Classic MWAA runs a provisioned environment that is billed while it runs; Amazon MWAA Serverless (launched November 2025) runs workflows with pay-per-use compute and no environment to keep up. Either way, Airflow is more capability — and more to learn and operate — than a single scheduled API call. Orchestration in depth belongs to Task 1.3.

Event triggers: S3 Event Notifications vs EventBridge

Event triggers make pipelines react when data arrives instead of polling for it.

The overlap rule. Two notification configurations on a bucket may not overlap: same event type (including wildcards — s3:ObjectCreated:* overlaps s3:ObjectCreated:Put) and prefix/suffix filters that could both match one key. A configuration with no suffix filter overlaps every suffix-filtered configuration on the same prefix and event type, so adding a suffix to only one of them does not remove the error. To send one event to several consumers, publish it to an SNS topic that fans out to SQS queues, or turn on EventBridge for the bucket.

Delivery is at least once. S3 notifications typically arrive within seconds, but an event can occasionally be delivered more than once, and not necessarily in order. Each record carries a sequencer value that orders events for the same key. Consumers must therefore be idempotent — for example, skip an object already processed.

Prefer event triggers over polling schedules when files arrive unpredictably: you avoid both waiting for the next scheduled run and the wasted runs that find nothing.

Lambda consumes a Kinesis stream through an event source mapping (ESM): Lambda polls the shards for you and invokes your function synchronously with batches of records, in order per shard, and tuning that mapping — throughput, batching, start position, filtering and error handling — is a core DEA-C01 skill.

Throughput and batching

By default one batch per shard is processed at a time. The parallelization factor (1–10) allows up to 10 concurrent batches per shard while still keeping order per partition key, so the mapping's concurrency is shards × parallelization factor — a 6-shard stream at factor 3 runs at most 18 invocations. Reserved concurrency on the function can only cap that number, never raise it; to go faster, raise the factor or add shards.

Lambda sends a batch when one of three things happens: the batch reaches batch size, the batching window expires, or the payload reaches 6 MB. For streams the window defaults to 0, so Lambda invokes as soon as records are available — at low record rates every batch is small, and raising batch size alone changes nothing. To get fuller batches (fewer invocations), set a batching window (up to 300 seconds) as well.

Where reading starts, and what gets read

The starting position applies when the mapping is created: LATEST reads only records written from now on, TRIM_HORIZON starts at the oldest record still retained, and AT_TIMESTAMP starts at a given time. A new consumer that must process the backlog already in the stream needs TRIM_HORIZON (or a timestamp); LATEST skips it.

Filter criteria on the mapping drop irrelevant records before any invocation, so you pay nothing for them. For Kinesis, a JSON payload is matched on the data property — for example, a pattern on data.eventType. Filtering in the function code instead still invokes (and bills) for every record.

Error handling

A failed batch is retried and blocks that shard until it succeeds or the records expire — by default retries are unlimited and maximum record age is the stream's retention, so one poison record can stall a shard for days. The tools:

Do not confuse the ESM's on-failure destination with a function's dead-letter queue: a function DLQ (or an asynchronous-invocation destination) applies only to asynchronous invocations, and stream mappings invoke synchronously, so it never receives failed stream batches.

Putting it together. A mapping that must tolerate bad data typically combines four settings: bisect-on-error with a bounded retry count, so a malformed record is isolated and then skipped instead of blocking its shard; a maximum record age, so a backlog does not grow without limit; an on-failure destination (S3 when the payload itself must be kept for repair); and partial batch responses, so records that already succeeded are not reprocessed.

Throttling means a service is rejecting requests above its capacity, and the fix depends on which limit you hit — a table partition, a shard, a connection pool, or a downstream API — so the exam tests service-specific remedies for DynamoDB, Kinesis, Amazon RDS and rate-limited APIs.

DynamoDB

DynamoDB returns ProvisionedThroughputExceededException when reads or writes exceed table or partition capacity, in both capacity modes. Each partition serves at most 3,000 RCU and 1,000 WCU, and that limit applies in on-demand mode too — so a hot partition key throttles even when total table capacity is ample, and adding capacity does not help. Fix it with key design: spread writes for a hot key with write sharding (a random or calculated suffix such as key#0…key#9) and read the shards back together.

For bursts, know what each remedy does:

• Retries with exponential backoff and jitter — the AWS SDKs do this automatically; it smooths brief spikes.
• Auto scaling adjusts provisioned capacity from CloudWatch alarms, which takes minutes — too slow for a sudden burst, which throttles before it reacts.
• On-demand mode handles up to double the previous peak instantly — the simple answer for spiky, unpredictable traffic when the table may change mode.
• Buffer writes with SQS when the table must stay provisioned: producers send to a queue and a consumer (for example, Lambda with a capped concurrency) drains it at a rate the table can take.
• DAX caches reads. It is a write-through cache — every write still goes to the table — so it does nothing for write throttling.

Kinesis

Write throttling (ProvisionedThroughputExceededException on put) means a shard's 1 MB/s or 1,000 records/s limit is exceeded. All records with one partition key go to one shard, so a single key can never exceed one shard's limit — and a low-cardinality key leaves some shards hot while others idle. Use a high-cardinality key, or add a random suffix to a hot key when per-key ordering is not needed; then add shards or switch to on-demand mode for overall volume. Read throttling is covered under fan-out below.

Amazon RDS

RDS limits usually surface as connection exhaustion — classically hundreds of concurrent Lambda invocations each opening a database connection. Amazon RDS Proxy pools and reuses connections so the database sees a small, stable number; read replicas offload read-heavy queries. Batch reads instead of making row-at-a-time calls.

Protecting a rate-limited downstream

When Lambda reads from SQS and calls an API that allows only N concurrent requests, set the SQS event source mapping's maximum concurrency (2–1,000) to N. Reserved concurrency is the wrong tool here: Lambda's pollers keep pulling messages, invocations beyond the limit are throttled, the messages return to the queue, and after enough receives they land in the dead-letter queue. Maximum concurrency stops the poller pulling more than the function may process. For APIs called from your own code, apply the client-side pacing and backoff described in the API section.

IP allowlists

An IP allowlist admits connections only from known addresses or security groups (network ACLs can also restrict a subnet by CIDR), so you must know which address each AWS service presents:

• Fixed outbound IP for Lambda or Glue. Attach the function or Glue connection to private subnets whose route goes through a NAT gateway with an Elastic IP; the partner allowlists that one address. Lambda outside a VPC, or in a VPC without NAT, has no stable public IP.
• Glue connections in a VPC. Glue places ENIs in your subnet; their security group needs a self-referencing inbound rule (all TCP from the same security group) so Glue's workers can talk to each other, and the database's security group must allow that group. Because the ENIs have no public IP, the subnet needs an S3 gateway endpoint (or NAT) for Glue to reach S3.
• Firehose to a publicly accessible Redshift cluster connects from AWS-owned addresses, so the cluster's security group must allow the published Firehose CIDR block for that Region; the IAM role and credentials alone are not enough.

Fan-in is many producers writing into one stream; fan-out is one stream feeding many consumers — and managing both without throttling is an explicit DEA-C01 skill.

Fan-in

Thousands of devices or services writing tiny records exhaust a shard's records-per-second limit long before its megabyte limit. Two tools solve different halves of that problem:

• KPL aggregation (Kinesis Producer Library) packs many small user records into one Kinesis record, so the shard counts fewer records against its 1,000 records/s limit. Consumers de-aggregate with the KCL or the Lambda/Firehose support for KPL records.
• PutRecords sends up to 500 records in one API call. It cuts per-request overhead and latency for the producer, but every record still counts individually against the shard limits — batching calls does not raise per-shard throughput.

Pair either with a partition key that spreads writes evenly. A common fan-in source is logs from many accounts: a CloudWatch Logs subscription filter in each sending account can stream to a CloudWatch Logs destination in the central account, which wraps a Kinesis data stream (or Firehose stream) there. The destination's access policy lists the sending accounts, and an IAM role lets CloudWatch Logs write to the stream.

Fan-out

Standard Kinesis consumers share each shard's 2 MB/s read throughput and poll it with GetRecords, which is limited to five calls per second per shard across all of them. Each extra polling consumer therefore waits longer between reads, so latency rises; adding shards raises total read capacity, but every polling consumer still shares each shard's 2 MB/s and five GetRecords calls per second. Enhanced fan-out gives each registered consumer its own dedicated 2 MB/s per shard, pushed over HTTP/2 (typically about 70 ms latency), for an extra charge. Rising read latency or ReadProvisionedThroughputExceeded as consumers are added is the symptom of that sharing, and dedicated per-consumer throughput is what removes it.

For message-based distribution, the SNS-to-SQS fan-out pattern delivers each published event to several independent queues, each consumed at its own pace with its own retries. On MSK, fan-out is native: each Kafka consumer group receives the full topic independently.

Fan-out also decouples: one stream can feed Firehose (raw archive to S3 — Firehose buffers and delivers in batches and does not guarantee record order, so do not build order-dependent consumers on its output), a Lambda alerting function and a Managed Service for Apache Flink application, each isolated from the others' failures.

A replayable ingestion pipeline can reprocess past data after a bug or outage, and replayability comes from sources that retain records after delivery.

Kinesis Data Streams keeps records for its configured retention — 24 hours by default, extendable to 7 days and, with long-term retention, up to 365 days — regardless of consumption, so a consumer can rewind to a sequence number or timestamp. Kafka on Amazon MSK is equally replayable through topic retention and consumer offsets. By contrast, SQS deletes a message once it is processed, SNS keeps nothing, and Firehose delivers and moves on — none of them can replay.

Retention costs money: extended and long-term retention are billed per shard-hour and per GB stored. If replays are rare and can be slower, archiving raw events to S3 (for example, a Firehose consumer of the stream) is far cheaper than keeping a year in the stream, and any batch engine can reprocess it. When Firehose runs a Lambda transformation, turn on source record backup so the untransformed records are also written to S3 — otherwise a buggy transform leaves only its output, with nothing raw to replay. Replay makes duplicates inevitable, so downstream processing must be idempotent.

Stateless vs stateful

A stateless operation handles each record on its own — validate, mask a field, convert format, route — so it scales horizontally without coordination and suits Lambda. A stateful operation needs memory across records: windowed aggregations (orders per minute), deduplication, sessionization and stream joins. That state must live somewhere durable:

Spark Structured Streaming (Glue streaming jobs) also checkpoints state, to S3. If a question involves windows, sessions or joins across events, plain stateless Lambda is the wrong tool; pick Lambda tumbling windows only for small, short, per-shard aggregates.