Orchestrating Data Pipelines on AWS: Step Functions, MWAA, and Glue Workflows

An orchestrator exists because real pipelines are multi-step, interdependent, and failure-prone: a raw file lands in Amazon S3, a crawler updates the schema, a transform job runs, a load into Amazon Redshift follows, and a data-quality check gates the whole thing. Without orchestration, you glue these together with cron jobs and hope — no retries, no dependency ordering, no visibility into which step failed.

An orchestration service gives you four things the exam repeatedly probes. Dependency management: step B runs only after step A succeeds, and parallel branches run concurrently to cut wall-clock time. Failure handling: automatic retries with backoff, error-specific catch paths, and fallback logic. State: the orchestrator remembers where a run is, so a resumed or replayed run does not redo completed work. Observability: a visual or logged execution history that shows exactly which step failed and why.

On AWS, the orchestration candidates are AWS Step Functions (serverless state machines), Amazon MWAA (managed Apache Airflow), and AWS Glue workflows (Glue-native orchestration), with Amazon EventBridge supplying schedules and event triggers and AWS Lambda handling small custom steps. Amazon SNS and Amazon SQS handle notifications and decoupling. The exam rarely asks you to define these services — it asks you to pick one under constraints such as least operational overhead, existing Airflow DAGs, or a Glue-only pipeline. Learn the selection logic, not just the feature lists.

AWS Step Functions is the default serverless orchestrator for ETL on AWS: you define a state machine in Amazon States Language (JSON), and Step Functions runs it with no infrastructure to manage, per-state retry policies, and direct integrations with over two hundred AWS services — including AWS Glue, Amazon EMR, AWS Lambda, Amazon Athena, and Amazon ECS — so most steps need no wrapper code at all. Its building blocks are Task states (do work), Choice (branch), Parallel (run fixed branches concurrently), Map (iterate over items), Wait, and the terminal Succeed/Fail states.

Standard or Express is a creation-time decision. The two workflow types differ in duration, execution semantics, history and features:

Exactly-once matters when a step has a non-idempotent side effect (charging, emailing, appending): at-least-once Express may run a step twice, so either make the step idempotent or use Standard. Use Standard for long-running ETL that runs Glue or EMR jobs or waits for people; use Express for high-volume, short event processing, or for an API that needs the result synchronously. Picking Express for a two-hour Glue pipeline is a classic trap: it times out at five minutes and cannot wait on a job with .sync anyway.

The type cannot be changed after a state machine is created; moving a workflow to Express means creating a new state machine from the same definition. The two can be nested: a Standard parent can start Express child executions for high-volume short steps, and an Express workflow that needs a long or human wait can hand that part to a Standard child.

A Step Functions task calls a service in one of three patterns, and the right one depends on whether the workflow must wait and who reports completion.

.sync only works for the AWS services Step Functions has run-a-job support for; it cannot watch an on-premises scheduler or a SaaS job. For those, send the task token to the external system (for example through an SQS message a worker reads) and have it call back. Neither .sync nor .waitForTaskToken is available in Express workflows.

Detecting a dead worker. A callback task can wait up to the execution's maximum, so set TimeoutSeconds for the overall limit and HeartbeatSeconds for liveness: the worker calls SendTaskHeartbeat periodically, and if no heartbeat arrives within the interval the task fails with States.HeartbeatTimeout — long before the overall timeout (States.Timeout) would fire.

IAM for .sync. For ECS and AWS Batch (and some other) .sync integrations, Step Functions tracks job completion through a managed EventBridge rule, so the execution role needs events:PutTargets, events:PutRule and events:DescribeRule on that rule (for ECS, StepFunctionsGetEventsForECSTaskRule) in addition to permission to run the task itself.

Step Functions handles errors with two per-state fields: Retry re-runs the state for transient errors, and Catch routes to a fallback state once retries are exhausted or for errors that should not be retried.

Retriers. Each retrier names the errors it handles in ErrorEquals and sets IntervalSeconds (first wait), BackoffRate (multiplier, default 2.0), MaxAttempts, and optionally MaxDelaySeconds (cap on any wait) and JitterStrategy. A BackoffRate of 1.0 gives a constant interval; 2.0 with a 2-second start gives 2, 4, 8, 16… seconds. JitterStrategy: FULL randomizes each wait between zero and the computed delay, so many executions that failed together (for example on throttling) do not retry in lockstep. Retriers are evaluated in order and the first match wins; States.ALL matches every error and must appear alone in the last retrier. States.TaskFailed matches every error except States.Timeout. Scope retries narrowly: retry throttling and service exceptions, but let permanent errors (bad input, access denied) fail fast instead of burning attempts.

Catchers and ResultPath. A catcher sends the execution to a Next state — cleanup, compensation, an SNS alert, then a Fail state. In JSONPath mode its ResultPath decides what the next state receives:

Use "$.error" when the cleanup step needs both the file key from the input and the failure reason.

Parallel failure semantics. If any branch of a Parallel state fails without handling the error, the whole Parallel state fails and Step Functions stops the other branches. If sibling branches must finish even when one fails, catch errors inside each branch (so the branch ends successfully with an error marker) rather than only on the Parallel state.

The Map state runs the same steps for every item in a collection, in one of two modes, and the size of the job decides which.

Choose Distributed Map when the item list is large (millions of S3 objects), would exceed the payload or history quotas, or needs high parallelism. ItemBatcher (MaxItemsPerBatch, or a byte limit) groups items so each child execution or Lambda invocation processes a batch rather than one item — far fewer executions and invocations for small items. Use MaxConcurrency to protect a downstream system with limited capacity.

Amazon MWAA (Managed Workflows for Apache Airflow) is the choice when the requirement mentions Apache Airflow, existing DAGs, Python-defined workflows, or portability across clouds. AWS manages the Airflow web server, scheduler, and workers — patching, scaling workers within the range you configure, and integrating with IAM and CloudWatch — while you supply the code in an S3 bucket.

Airflow's strength is expressiveness: DAGs are Python code, so dynamic pipeline generation, complex branching, backfills, and a huge ecosystem of community operators (for AWS services, databases, SaaS APIs, and other clouds) come naturally. Teams migrating an on-premises Airflow deployment or orchestrating steps that span non-AWS systems land here. The scheduler-based model also gives rich calendar semantics — data intervals, catchup runs, and per-task SLAs — that Step Functions does not provide natively.

The environment's S3 layout. An MWAA environment reads DAG files from a dags/ folder, custom operators and hooks from a plugins.zip, Python dependencies from a requirements.txt, and an optional shell startup script that runs on each component before Airflow starts (environment variables, OS-level installs). Changing plugins or requirements means pointing the environment at the new object version and updating it; dependency conflicts in requirements.txt are a common cause of failed updates.

Capacity levers.

Trade-offs. A provisioned MWAA environment bills for every hour it runs, whether or not a DAG is active, and you still own DAG code quality, dependencies and Airflow version upgrades — so for a handful of simple AWS-native pipelines Step Functions is cheaper and lower-overhead. AWS also launched Amazon MWAA Serverless (November 2025), which runs workflows defined in YAML (AWS supplies a converter for existing Python DAGs) with automatic scaling and pay-per-use pricing, so "Airflow always means an idle environment bill" is no longer true in general. The selection rule: existing Airflow investment or need for Airflow's ecosystem → MWAA; greenfield AWS-native serverless orchestration → Step Functions.

AWS Glue workflows orchestrate pipelines whose every step is a Glue component — crawlers, Glue ETL jobs, and triggers — from within the Glue console, with no additional service to configure. A workflow is a graph of triggers: a schedule or on-demand trigger starts it, event triggers fire the next crawler or job when predecessors complete, and conditional triggers can require all (or any) of several predecessors to succeed. Workflow run properties let jobs in the same run share key-value state.

The typical pattern is exactly the exam's typical scenario: crawl a raw S3 prefix to update the AWS Glue Data Catalog, run an ETL job that converts CSV to Parquet and partitions the output, then crawl the curated prefix so downstream Amazon Athena queries see fresh partitions. If that is the whole pipeline, a Glue workflow is the answer with the least operational overhead — everything stays in one service, one console, one set of IAM permissions.

Know the boundary, because it is where questions pivot to Step Functions. Glue workflows can only orchestrate Glue crawlers and Glue jobs. The moment the pipeline must also invoke a Lambda function, wait for a Redshift load, branch on custom logic, call Amazon EMR, or pause for approval, Glue workflows cannot express it — Step Functions (which has native Glue integrations) takes over as the orchestrator and treats each Glue job as one task state. Also distinguish workflows from Glue job triggers used alone: triggers chain a couple of jobs, while a workflow gives you the full graph view and run-level monitoring.

Event-based starts and Glue alerts. A Glue workflow can start from an Amazon EventBridge event (for example S3 object-created events) with an event batching condition: the workflow starts when the batch size is reached (up to 100 events) or the batch window expires (up to 900 seconds), whichever comes first — so a burst of small files triggers one run instead of hundreds. For failure alerts, Glue emits Glue Job State Change events to EventBridge; a rule matching FAILED and TIMEOUT states that targets an SNS topic alerts on every failed run. The job's notification delay threshold is different: it raises an event when a run is still going after the given number of minutes — a "running late" signal, not a failure alert.

Amazon EventBridge is how pipelines start, both on a clock and on an event. The canonical data-engineering pattern: enable EventBridge notifications on an S3 bucket, write a rule matching Object Created for a specific prefix, and target a Step Functions state machine so the pipeline runs the moment data arrives instead of polling on a schedule.

S3 event notifications versus EventBridge. An S3 bucket's native event notifications deliver only to four destinations: an SNS topic, an SQS queue (standard), a Lambda function, or — when turned on — EventBridge, which then receives all the bucket's events. Step Functions is not a direct notification destination; you reach it through an EventBridge rule (or a Lambda function that calls StartExecution). Native notification configurations also cannot overlap: two configurations for the same event type with overlapping prefix/suffix filters are rejected. EventBridge rules have no such restriction — several rules can match the same objects and each can target a different pipeline — and support richer filtering on the event content.

Scheduled starts. Two options exist:

A flexible time window lets Scheduler invoke the target any time within a window after the scheduled time, spreading out many schedules that share a start time so they do not hit the target in one spike. Scheduler is the recommended choice for new schedules.

Event-driven beats schedule-driven when arrival times vary: a schedule either runs before the data lands (wasted run, or worse, processing yesterday's data) or long after (latency).

AWS Lambda plays two orchestration roles. As a step, it handles lightweight custom logic inside a workflow — validating a manifest, calling an external API, transforming a small payload — cheaper and faster to start than a Glue job for small work. As a trigger, a Lambda invoked by an event can start or signal a larger workflow. But chaining many Lambdas directly into each other as an orchestration mechanism is an anti-pattern: you lose central retry policy, state visibility, and timeout control. If a design describes Lambda-calls-Lambda spaghetti, the fix is to refactor into a Step Functions state machine.

How a failed Lambda event is retried and captured depends on how the function was invoked, and data pipelines use both styles.

Asynchronous invocation — used by S3 event notifications, SNS and EventBridge. Lambda queues the event and, if the function errors, retries it twice by default (with waits of about one and two minutes), then discards it unless something captures it. You tune this with MaximumRetryAttempts (0–2) and MaximumEventAgeInSeconds (60–21,600 seconds, default six hours) and capture failures with an on-failure destination (SQS, SNS, Lambda, EventBridge or S3), which records the event plus error details, or the older function dead-letter queue (SQS or SNS, event only). On-success destinations report successful results.

Event source mappings — Lambda polls a source (SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, self-managed Kafka) and invokes the function synchronously. The asynchronous retry settings, async destinations and function DLQ do not apply here. For SQS, failed messages return to the queue after the visibility timeout and the queue's own redrive policy moves them to a DLQ. On-failure destinations on the mapping itself exist only for stream and Kafka sources. Because S3 invokes Lambda asynchronously (there is no event source mapping), an S3-triggered function captures failures with an async on-failure destination or DLQ.

Idempotent writes. Retries and at-least-once delivery mean a function may process the same event twice, so make the effect repeatable: write to a deterministic S3 key derived from the input (a rerun overwrites the same object), and load warehouses with staging-table MERGE upserts rather than blind inserts. S3 Versioning does not deduplicate — it keeps every write to a key as another version — so idempotency comes from deterministic keys, not from versioning.

Choose by matching the pipeline's shape and constraints to each service's sweet spot — the table below is the comparison the exam draws on.

Exam heuristics: least operational overhead or serverless → Step Functions (or a Glue workflow if the pipeline is Glue-only); existing Airflow / open-source portability → MWAA; Glue crawlers and jobs only → Glue workflows; most cost-effective for infrequent runs → not a provisioned MWAA environment, because it bills while idle.

A resilient pipeline assumes every step can fail and makes failure cheap. Build that in four layers.

Retries with exponential backoff. Transient failures — throttling, brief service errors — should be retried automatically, with growing, jittered intervals and a capped attempt count, while permanent errors fail fast. In Step Functions this is the per-state Retry field (see the error-handling section); in Airflow it is per-task retries and delay settings.

Idempotency and replayability. Retries and replays are only safe if reprocessing the same input does not duplicate output. Design steps so reruns overwrite or upsert rather than append blindly — write to deterministic S3 keys, use staging-table merge patterns for Redshift loads, and key streaming writes so duplicates collapse. You should be able to rerun yesterday's partition without corrupting the dataset.

Decoupling with queues. Placing Amazon SQS between a producer and a processing stage buffers bursts, lets each side scale independently, and adds durability — if the consumer is down, messages wait. Two settings matter. The visibility timeout hides a message while a consumer works on it; if it is shorter than processing time, the message reappears and is processed twice. For a Lambda event source, AWS recommends a queue visibility timeout of at least six times the function timeout. The event source mapping's maximum concurrency caps how many concurrent function instances the queue can drive, so a burst drains steadily instead of overwhelming a fixed-capacity database. A dead-letter queue sets aside messages that repeatedly fail: configure a redrive policy on the source queue with maxReceiveCount (receives before moving), point it at a DLQ of the same type (standard or FIFO) in the same account and Region, and give the DLQ a longer retention period than the source, because a standard queue keeps the original enqueue timestamp when a message is moved. After fixing the bug, redrive the messages back to the source.

Failure isolation and scale. Use Catch branches to route errors to cleanup and alert states so one bad file fails its branch, not the whole run; a Distributed Map can tolerate a percentage of failed items and fan out to thousands of parallel children. Together these deliver the performance, availability, scalability, and fault-tolerance requirements Task 1.3 names.

Use Amazon SNS to tell humans and systems that something happened, and Amazon SQS to hand work between pipeline stages — they are complementary, not interchangeable. SNS is push-based publish/subscribe: one published message fans out to every subscriber (email, SMS, Lambda, SQS queues, HTTP/S endpoints). SQS is pull-based queuing: consumers take messages at their own pace, one consumer processes each message, with durability and retry semantics.

Fan-out versus competing consumers. Publish once to an SNS topic subscribed by one SQS queue per downstream system, so each gets its own durable copy and processes it independently. Several consumers reading one shared queue are competing consumers — each message goes to only one of them, which scales a single job but does not give every system a copy. A subscription filter policy on each subscription (matching message attributes or body) delivers only the messages that subscriber wants. When order and de-duplication matter, use an SNS FIFO topic with SQS FIFO queues: messages are ordered per message group and duplicates within the deduplication window are dropped (content-based deduplication is a FIFO-only feature).

Delivery behaviours compared. Asynchronous Lambda invocation retries a failed event twice within minutes, then discards it unless a destination or DLQ captures it. SNS delivery to HTTP/S endpoints follows a configurable delivery policy (number of retries and backoff, up to about an hour), with a subscription DLQ for messages that still fail. SQS is the option when a consumer must pull work at its own pace — messages wait in the queue for up to the retention period.

Alerting on pipeline failures. The orchestrator emits failure signals and SNS delivers them. Inside the workflow, a Catch branch ends in an SNS publish task with the error detail. Outside it, an EventBridge rule matches Step Functions Execution Status Change events with status FAILED, TIMED_OUT or ABORTED and targets an SNS topic — this also catches executions that die without reaching a catch state, and gives one alert per execution. Only Standard workflows emit these events; for Express, alert from CloudWatch metrics such as ExecutionsFailed (an alarm on that metric) or from CloudWatch Logs. A CloudWatch alarm watches a metric against a threshold over a period, so it signals "failures happened" rather than naming each failed execution. That makes alarms the right tool for threshold-based signals — a job running longer than usual, or an SQS queue backing up (an alarm on ApproximateAgeOfOldestMessage or ApproximateNumberOfMessagesVisible) — with SNS as the alarm action. Glue job state change events and MWAA task callbacks raise alerts the same way.

Scenario. A retailer lands supplier CSVs in S3 at unpredictable times. Requirements: process each file on arrival, convert to Parquet with a Glue job, load to Redshift, alert the data team on any failure — with the least operational overhead. Answer: S3 EventBridge notifications trigger a Step Functions Standard workflow; it runs the Glue job with the run-a-job pattern, then the Redshift load; every state has scoped retries with backoff; a Catch branch publishes the error to an SNS topic that emails the team. No servers, no polling, no idle cost — and each requirement maps to one native feature.

Maintaining a Step Functions workflow means releasing changes safely, recovering failed runs, and being able to see what each run did.

Versions and aliases. Publishing a version freezes an immutable snapshot of the definition and settings. An alias points to one version, or to two versions with weighted routing (for example 90/10) — callers start executions through the alias ARN, so you can canary a new version and shift or roll back traffic by updating the alias without changing any trigger.

Redrive. A failed, aborted or timed-out Standard execution can be redriven within 14 days of when it stopped: it resumes from the failed step, reusing the results of the steps that already succeeded. StartExecution always begins at the definition's start state — there is no start-at-step parameter, whichever version you run — so redrive is the only way to resume mid-workflow (otherwise you design skip logic into the input yourself).

Observability.