Automate Data Processing on AWS: Step Functions, MWAA, Lambda & EventBridge

AWS gives you two managed orchestrators for data pipelines: AWS Step Functions, a serverless state machine service, and Amazon MWAA (Managed Workflows for Apache Airflow), a managed deployment of open-source Airflow — and DEA-C01 expects you to know when each one is the better fit.

Step Functions defines workflows as state machines in Amazon States Language (JSON). It is serverless — no environment to run or pay for while idle — and it integrates directly with the data services you orchestrate: it can start a Glue job, an EMR step, an Athena query, an ECS task, or a Lambda function as a native state, and the run-a-job integration pattern (often written as .sync) makes a state wait until the job actually finishes before moving on. Each state can declare its own Retry (with backoff) and Catch (error routing) behavior, which is how you build fault tolerance declaratively instead of writing polling loops. The Map state fans work out across items, and Distributed Map scales that pattern to very large collections such as millions of S3 objects. Step Functions comes in two types: Standard workflows run for up to a year with a full, inspectable execution history — the right choice for long-running ETL — while Express workflows run for at most five minutes at very high volume, suited to event-processing bursts rather than batch pipelines.

Amazon MWAA runs Apache Airflow for you: you author DAGs in Python, upload them to an S3 bucket the environment watches, and Airflow's scheduler and workers execute the tasks. Choose MWAA when the scenario describes complex dependency graphs, backfills over historical date ranges, a team that already writes Airflow DAGs, or orchestration that must reach beyond AWS through Airflow's large operator ecosystem (on-premises systems, other clouds, third-party tools). The trade-off is operational shape: an MWAA environment is always on and billed while it exists, so for a purely AWS-native pipeline with a handful of steps, Step Functions is almost always the cheaper, lower-overhead answer.

A third, narrower option is AWS Glue workflows: chains of Glue crawlers and Glue jobs connected by triggers (scheduled, on-demand, conditional, or EventBridge-event based). If everything in the pipeline is a Glue crawler or Glue job, a Glue workflow keeps orchestration inside one service with nothing extra to build — but the moment the pipeline needs Lambda, Athena, EMR, or human-style branching, you have outgrown it.

Match the automation service to the workflow's complexity and trigger style — the exam signals the intended answer with phrases like least operational overhead, existing Airflow DAGs, or only Glue jobs and crawlers.

Two gotchas the exam likes. First, EventBridge schedules things; it does not orchestrate them — it can start a Glue job at midnight, but it cannot wait for the job, check the result, and start the next step. If the question involves sequencing with failure handling, the answer is Step Functions or MWAA, with EventBridge merely providing the kickoff. Second, do not chain Lambda functions by having one invoke the next: that hand-rolled orchestration loses retries, visibility, and state, and the exam-correct refactor is Step Functions.

Troubleshooting a managed workflow starts with the service's own execution record: MWAA writes component logs to Amazon CloudWatch, Step Functions records every state transition in its execution history, and Glue workflows expose the run state of each node.

MWAA problems cluster into three patterns. A DAG that never appears in the Airflow UI usually means a Python import error in the DAG file or a file uploaded outside the configured S3 DAGs prefix — check the scheduler log group in CloudWatch for the parse error. Tasks stuck in the queued state point to worker capacity: the environment's workers are saturated, so increase the maximum worker count or move to a larger environment class. Dependency failures come from requirements.txt: MWAA installs the packages you list at environment startup, and an incompatible version pin can break workers — pin versions deliberately and test changes in a staging environment. MWAA emits separate CloudWatch log groups for the scheduler, workers, web server, and task logs; knowing which log group answers which symptom is exactly the kind of detail Task 3.1 tests.

Step Functions debugging is anchored on the execution history, which shows the input, output, and error of every state. Common failure names matter: States.TaskFailed wraps an error thrown by the invoked service, States.Timeout means a state exceeded its configured timeout, and a data-size error means the payload passed between states outgrew the service's limit — the standard fix is to write large intermediate data to Amazon S3 and pass only the object key between states, never the data itself. If a downstream state received the wrong input, inspect the state's input/output processing (paths and selectors) in the history rather than the worker code.

Glue workflows stall when a conditional trigger's condition is never met — for example, a predecessor job failed, so the trigger that waits for its success never fires. The workflow run view shows which node stopped the chain; the job's own CloudWatch logs and error output explain why. Also remember that a job retried after fixing bad input may need attention to its job bookmark: bookmarks track what was already processed, so reprocessing corrected data can require resetting or rewinding the bookmark.

Pipeline code reaches AWS features through the SDKs — boto3 in Python being the one you will see most — and the exam tests the habits that make SDK calls production-safe rather than the syntax.

Credentials come from IAM roles, never from code. A Lambda function, Glue job, or EMR step gets temporary credentials from its execution role automatically; the SDK picks them up from the environment with no keys embedded anywhere. Retries are built in: the SDKs retry throttled and transient failures with exponential backoff by default, and you tune behavior through retry modes (standard or adaptive) rather than writing your own retry loops. Pagination is explicit: list operations return one page of results plus a continuation token, and forgetting this is a classic bug — a script that lists S3 objects and silently sees only the first page. Use the SDK's paginators to iterate every page. Waiters poll a resource until it reaches a state (a cluster becomes available, a table becomes active) so you do not hand-roll sleep loops.

Many data-service APIs are asynchronous: StartJobRun (Glue), StartQueryExecution (Athena), and ExecuteStatement (Redshift Data API) return immediately with an identifier, and your code either polls a status call or — better — lets EventBridge deliver the completion event. The Redshift Data API deserves special attention: it lets Lambda, Step Functions, and notebooks run SQL against Redshift over HTTPS with IAM authentication, no JDBC driver, no persistent connection, and no VPC networking from the caller's side. When a question asks how a Lambda function should query Redshift without managing database connections, the Data API is the answer.

The anti-pattern to recognize: a Lambda function that calls StartQueryExecution and then sleeps in a loop waiting for the result is burning paid compute on waiting. The exam-correct designs are a Step Functions state machine that polls between states, the run-a-job integration that waits natively, or an EventBridge rule on the completion event.

Before adding orchestration code, use the automation features the processing services already ship — DEA-C01 rewards answers that let Glue, EMR, and Redshift do work natively instead of bolting on custom infrastructure.

AWS Glue automates incrementality with job bookmarks (each run processes only data added since the last successful run), chains work with triggers and workflows, scales with auto scaling worker fleets, and offers Flex execution for non-urgent jobs at lower cost. Glue streaming jobs run Spark Structured Streaming continuously against Kinesis or Kafka when the transformation itself must be always-on. Crawlers keep the Data Catalog's schemas and partitions synchronized as new data lands, which is itself automation — downstream Athena queries see new partitions without manual DDL.

Amazon EMR processes at cluster scale, and its automation primitive is the step: you submit Spark or Hive steps to a cluster through the API, and a transient cluster can be configured to terminate automatically after its steps complete — the classic cost-controlled batch pattern, usually driven by Step Functions. Managed scaling resizes long-running clusters with load, and EMR Serverless removes cluster management entirely: you submit jobs to an application and pay for the resources the job actually uses, the right answer when the scenario wants Spark without capacity planning.

Amazon Redshift automates transformation inside the warehouse: stored procedures encapsulate multi-statement SQL transformations, scheduled queries run them on a cron (driven by EventBridge and the Data API under the hood), materialized views with auto refresh keep precomputed aggregations current, and auto-copy from S3 loads new files as they arrive without an external trigger. When a question describes recurring SQL-shaped transformation on data already in Redshift, the ELT answer — run it in Redshift on a schedule — beats exporting the data to an external processing service and loading it back.

Preparing data for transformation means profiling it, fixing quality issues, and standardizing formats before the heavy processing runs — and the two services the exam names for this are AWS Glue DataBrew and Amazon SageMaker Unified Studio.

DataBrew is visual, no-code data preparation. You open a project on a sample of a dataset (from S3, the Glue Data Catalog, Redshift, or JDBC sources), build a recipe from its library of more than 250 built-in transformations — remove or fill nulls, standardize casing and date formats, split and merge columns, deduplicate, handle outliers — and then run the recipe as a recipe job against the full dataset on a schedule, writing cleaned output to S3. Profile jobs generate statistics about a dataset (completeness, distributions, correlations, duplicates), and data quality rules validate datasets against thresholds you define. The exam signal for DataBrew is a scenario where analysts or less-technical users must clean and normalize data without writing code; if the transformation needs custom logic or joins at scale, a Glue Spark job is the better fit.

SageMaker Unified Studio is the single environment that brings AWS analytics and machine learning tools together: within a governed project, teams share access to data through the lakehouse architecture, prepare it with visual ETL flows or JupyterLab notebooks, query it with built-in SQL editors, and publish assets to a business catalog for discovery. For Task 3.1, its relevance is collaborative preparation: instead of each engineer stitching together consoles, a project gives data engineers, analysts, and scientists one governed place where prepared datasets, queries, and notebooks live together. When a scenario emphasizes a unified, project-based environment spanning data preparation through analytics and ML, Unified Studio is the answer.

Positioning matters on the exam: DataBrew is the no-code cleanup tool, Glue ETL is the code-first Spark engine, and Unified Studio is the umbrella workspace where preparation happens alongside querying and model work — they complement rather than replace each other.

Amazon Athena is the serverless query layer of an automated pipeline: it runs standard SQL directly against data in S3 using Glue Data Catalog schemas, with no infrastructure and pricing based on data scanned. Inside pipelines it appears two ways. First, as an automated step: Step Functions can run an Athena query natively, and code calls StartQueryExecution and reacts to completion — used for validation queries, report extracts, or transformations expressed as CTAS (CREATE TABLE AS SELECT), which materializes query results back to S3 in a columnar format like Parquet, partitioned and compressed. Second, as the verification tool: after a load, an Athena query that counts rows or checks for nulls is the cheapest automated quality gate. Cost and performance discipline carries the marks: partition the data and use columnar formats so queries scan less, and use workgroups to separate teams, enforce per-query scan limits, and route results to controlled S3 locations.

Consuming a data API means calling someone else's interface politely: authenticate as required (API keys, IAM SigV4, or OAuth tokens), page through results using the API's continuation tokens, respect rate limits by backing off with exponential delay and jitter when throttled, and treat retries as inevitable — which means the consuming pipeline must be idempotent so a retried page never double-lands data. A Lambda function on an EventBridge schedule that pulls an external API and writes raw responses to S3 is the canonical consumption pattern.

Maintaining a data API is the other direction: exposing your data to other systems. The standard AWS build is Amazon API Gateway in front of Lambda, which queries the backing store (DynamoDB for key-value lookups, Athena or Redshift for analytical reads). Maintenance is what the exam probes: usage plans and throttling protect the backend from bursty consumers, caching at the API layer absorbs repeated reads, versioning (a version in the path or a stage per version) lets the schema evolve without breaking existing consumers, and monitoring of latency and error rates tells you when the backend store no longer fits the access pattern.

AWS Lambda is the automation glue of Task 3.1: it runs code in response to events with no servers, which makes it the default executor for lightweight processing — validating and tagging a file the moment it lands in S3, transforming small payloads, invoking service APIs, and dispatching notifications. Its boundaries define its exam role: execution time is capped at 15 minutes and resources at function scale, so Lambda is the wrong answer for heavy transformation of large datasets — that work belongs in Glue or EMR, with Lambda merely starting it. Concurrency is the other lever: hundreds of S3 events can invoke hundreds of parallel executions, which is powerful for fan-out but demands idempotent handlers and awareness of downstream limits (a database receiving hundreds of simultaneous connections, for instance).

Amazon EventBridge supplies both halves of triggering. Rules pattern-match events on the event bus and route them to targets: S3 object-created events (delivered via EventBridge) can start a Step Functions execution with rich filtering on bucket and key; a Glue job state change event with state FAILED can route to an SNS topic for alerting or to a remediation Lambda — the standard answer for reacting to pipeline failures without polling. EventBridge Scheduler handles time: cron and rate schedules, time-zone awareness, one-time schedules, and flexible time windows, invoking nearly any AWS API as its target. EventBridge can also archive and replay events, which gives event-driven pipelines a recovery path after a downstream bug.

Scenario

A retail company receives supplier inventory files in S3 throughout the night and needs them validated, transformed to Parquet, made queryable, and reported on by 7 AM — with alerts on any failure and no idle infrastructure. The exam-shaped design: S3 events (via EventBridge) trigger a Step Functions Standard workflow per file batch; the state machine runs a Lambda validation step, then a Glue job (run-a-job pattern, so the workflow waits) to convert and partition the data, then a Glue crawler to register new partitions, then an Athena CTAS query for the morning aggregate. Every state has Retry with backoff; a Catch route publishes failures to SNS. An EventBridge Scheduler cron fires a final 6 AM completeness check. Nothing runs — or bills — between files, and every failure is visible in one execution history.