Orchestrating Data Pipelines on AWS: Step Functions, MWAA, and Glue Workflows

An orchestrator exists because real pipelines are multi-step, interdependent, and failure-prone: a raw file lands in Amazon S3, a crawler updates the schema, a transform job runs, a load into Amazon Redshift follows, and a data-quality check gates the whole thing. Without orchestration, you glue these together with cron jobs and hope — no retries, no dependency ordering, no visibility into which step failed.

An orchestration service gives you four things the exam repeatedly probes. Dependency management: step B runs only after step A succeeds, and parallel branches run concurrently to cut wall-clock time. Failure handling: automatic retries with backoff, error-specific catch paths, and fallback logic. State: the orchestrator remembers where a run is, so a resumed or replayed run does not redo completed work. Observability: a visual or logged execution history that shows exactly which step failed and why.

On AWS, the orchestration candidates are AWS Step Functions (serverless state machines), Amazon MWAA (managed Apache Airflow), and AWS Glue workflows (Glue-native orchestration), with Amazon EventBridge supplying schedules and event triggers and AWS Lambda handling small custom steps. Amazon SNS and Amazon SQS handle notifications and decoupling. The exam rarely asks you to define these services — it asks you to pick one under constraints such as least operational overhead, existing Airflow DAGs, or a Glue-only pipeline. Learn the selection logic, not just the feature lists.

AWS Step Functions is the default serverless orchestrator for ETL on AWS: you define a state machine in Amazon States Language (JSON), and Step Functions runs it with no infrastructure to manage, per-state retry policies, and direct integrations with over two hundred AWS services — including AWS Glue, Amazon EMR, AWS Lambda, Amazon Athena, and Amazon ECS — so most steps need no wrapper code at all.

Three state-machine features matter most for data engineering. Service integration patterns: the .sync (run-a-job) pattern starts a Glue job or EMR step and waits for it to finish before moving on, which is exactly what sequential ETL needs; the wait-for-callback pattern (task token) pauses the workflow until an external system calls back, useful for human approval or long-running external processes. Error handling: every task state supports Retry (with interval, backoff rate, and max attempts) and Catch (route specific error types to a cleanup or alert branch). Fan-out: a Map state iterates over a collection — in distributed mode it can process very large datasets, such as iterating over every object in an S3 prefix, with massive parallelism.

Choose between the two workflow types deliberately. Standard workflows support long-running executions (up to a year), exactly-once state transitions, and full execution history — the right choice for nightly ETL that runs Glue or EMR jobs. Express workflows are for high-volume, short-lived executions (up to five minutes) with at-least-once semantics and lower cost per execution — the right choice for streaming-adjacent event processing. Picking Express for a two-hour Glue pipeline is a classic exam trap: it will time out.

Amazon MWAA (Managed Workflows for Apache Airflow) is the choice when the requirement mentions Apache Airflow, existing DAGs, Python-defined workflows, or portability across clouds. AWS manages the Airflow web server, scheduler, and workers — patching, scaling workers within the range you configure, and integrating with IAM and CloudWatch — while you supply DAG files, plugins, and a requirements.txt via an S3 bucket.

Airflow's strength is expressiveness: DAGs are Python code, so dynamic pipeline generation, complex branching, backfills, and a huge ecosystem of community operators (for AWS services, databases, SaaS APIs, and other clouds) come naturally. Teams migrating an on-premises Airflow deployment or orchestrating steps that span non-AWS systems land here. The scheduler-based model also gives you rich calendar semantics — data intervals, catchup runs, and per-task SLAs — that Step Functions does not provide natively.

The trade-offs are the exam's favorite angle. MWAA is not serverless: an environment runs continuously and bills per hour whether or not any DAG is running, so for a handful of simple pipelines it is neither the most cost-effective option nor the least operational overhead — Step Functions wins both. You still own DAG code quality, dependency conflicts in requirements.txt, and Airflow version upgrades. Workers scale within configured bounds rather than instantly to any level. The selection rule to memorize: existing Airflow investment or need for Airflow's ecosystem → MWAA; greenfield AWS-native serverless orchestration → Step Functions.

AWS Glue workflows orchestrate pipelines whose every step is a Glue component — crawlers, Glue ETL jobs, and triggers — from within the Glue console, with no additional service to configure. A workflow is a graph of triggers: a schedule or on-demand trigger starts it, event triggers fire the next crawler or job when predecessors complete, and conditional triggers can require all (or any) of several predecessors to succeed. Workflow run properties let jobs in the same run share key-value state.

The typical pattern is exactly the exam's typical scenario: crawl a raw S3 prefix to update the AWS Glue Data Catalog, run an ETL job that converts CSV to Parquet and partitions the output, then crawl the curated prefix so downstream Amazon Athena queries see fresh partitions. If that is the whole pipeline, a Glue workflow is the answer with the least operational overhead — everything stays in one service, one console, one set of IAM permissions.

Know the boundary, because it is where questions pivot to Step Functions. Glue workflows can only orchestrate Glue crawlers and Glue jobs. The moment the pipeline must also invoke a Lambda function, wait for a Redshift load, branch on custom logic, call Amazon EMR, or pause for approval, Glue workflows cannot express it — Step Functions (which has native Glue integrations) takes over as the orchestrator and treats each Glue job as one task state. Also distinguish workflows from Glue job triggers used alone: triggers chain a couple of jobs, while a workflow gives you the full graph view and run-level monitoring.

Amazon EventBridge is how pipelines start. It covers both trigger styles the guide names: time-based schedules — cron or rate expressions that start a Step Functions execution, Glue workflow, or Lambda function at fixed times (EventBridge Scheduler is the newer, more scalable scheduling capability) — and event-driven triggers, where a rule matches an event pattern and routes it to a target. The canonical data-engineering pattern: enable EventBridge notifications on an S3 bucket, write a rule matching Object Created for a specific prefix, and target a Step Functions state machine so the pipeline runs the moment data arrives instead of polling on a schedule.

Event-driven beats schedule-driven when arrival times vary: a schedule either runs before the data lands (wasted run, or worse, processing yesterday's data) or long after (latency). When a question contrasts "process files as soon as they arrive" with a cron-style design, the event-driven answer is almost always correct.

AWS Lambda plays two orchestration roles. As a step, it handles lightweight custom logic inside a workflow — validating a manifest, calling an external API, transforming a small payload — cheaper and faster to start than a Glue job for small work. As a glue trigger, a Lambda invoked by an event can start or signal a larger workflow. But chaining many Lambdas directly into each other as an orchestration mechanism is an anti-pattern: you lose central retry policy, state visibility, and timeout control. If the question describes Lambda-calls-Lambda spaghetti, the fix is to refactor into a Step Functions state machine.

Choose by matching the pipeline's shape and constraints to each service's sweet spot — the table below is the comparison the exam draws on.

Exam heuristics: least operational overhead or serverless → Step Functions (or a Glue workflow if the pipeline is Glue-only); existing Airflow / open-source portability → MWAA; Glue crawlers and jobs only → Glue workflows; most cost-effective for infrequent runs → never MWAA, because the environment bills while idle.

A resilient pipeline assumes every step can fail and makes failure cheap. Build that in four layers.

Retries with exponential backoff. Transient failures — throttling, brief service errors — should be retried automatically, with growing intervals and a capped attempt count. In Step Functions this is the Retry field per state, and you can retry different error types differently (retry a throttle, fail fast on a permissions error). In Airflow it is per-task retries and delay settings.

Idempotency and replayability. Retries and replays are only safe if reprocessing the same input does not duplicate output. Design steps so reruns overwrite or upsert rather than append blindly — write to deterministic S3 keys, use staging-table merge patterns for Redshift loads, and key streaming writes so duplicates collapse. The exam frames this as replayability of ingestion pipelines: you should be able to rerun yesterday's partition without corrupting the dataset.

Decoupling with queues. Placing Amazon SQS between a producer and a processing stage buffers bursts, lets each side scale independently, and adds durability — if the consumer is down, messages wait. Configure a dead-letter queue so messages that repeatedly fail processing are set aside for inspection instead of poisoning the pipeline with endless retries. Lambda event sources and SNS subscriptions support DLQs too.

Failure isolation. Use Catch branches to route errors to cleanup and alert states so one bad file fails its branch, not the whole run; a Map state can continue processing other items with a tolerated failure threshold. Together these deliver the performance, availability, scalability, and fault-tolerance requirements Task 1.3 names.

Use Amazon SNS to tell humans and systems that something happened, and Amazon SQS to hand work between pipeline stages — they are complementary, not interchangeable. SNS is push-based publish/subscribe: one published message fans out to every subscriber (email, SMS, Lambda, SQS queues, HTTPS endpoints). SQS is pull-based queuing: one consumer processes each message, with durability and retry semantics. The classic combined pattern is fan-out — publish once to an SNS topic subscribed by multiple SQS queues, so several downstream systems each get their own durable copy.

For pipeline alerting, the standard wiring is: the orchestrator emits failure signals, and SNS delivers them. Concretely, a Step Functions Catch branch ends in an SNS publish task with the error detail; or an EventBridge rule matches Step Functions execution-status-change events (FAILED, TIMED_OUT) and targets an SNS topic — which also catches executions that die without reaching a catch state. Glue job state changes and MWAA task callbacks can raise alerts the same way, and CloudWatch alarms on metrics (failed executions, job duration, queue depth) publish to SNS for threshold-based alerting.

Scenario. A retailer lands supplier CSVs in S3 at unpredictable times. Requirements: process each file on arrival, convert to Parquet with a Glue job, load to Redshift, alert the data team on any failure — with the least operational overhead. Answer: S3 EventBridge notifications trigger a Step Functions Standard workflow; it runs the Glue job with the run-a-job pattern, then the Redshift load; every state has retries with backoff; a Catch branch publishes the error to an SNS topic that emails the team. No servers, no polling, no idle cost — and each requirement maps to one native feature.