Automate Data Processing on AWS: Step Functions, MWAA, Lambda & EventBridge

AWS gives you two managed orchestrators for data pipelines: AWS Step Functions, a serverless state machine service, and Amazon MWAA (Managed Workflows for Apache Airflow), a managed deployment of open-source Airflow — and DEA-C01 expects you to know when each one is the better fit.

Step Functions defines workflows as state machines in Amazon States Language (JSON). It is serverless — no environment to run or pay for while idle — and it integrates directly with the data services you orchestrate: it can start a Glue job, an EMR step, an Athena query, an ECS task, or a Lambda function as a native state, and the run-a-job integration pattern (often written as .sync) makes a state wait until the job actually finishes before moving on. Each state can declare its own Retry (with backoff) and Catch (error routing) behavior, which is how you build fault tolerance declaratively instead of writing polling loops. The Map state fans work out across items, and Distributed Map scales that pattern to very large collections such as millions of S3 objects. Step Functions comes in two types, and they differ in duration, semantics and price:

Standard is the right choice for long-running ETL. Express suits short, very high-volume event processing — tens of thousands of quick validate-enrich-write runs an hour cost far less billed per request and duration than per transition — provided each step is idempotent, because an asynchronous Express execution can run a step more than once.

Amazon MWAA runs Apache Airflow for you: you author DAGs in Python, upload them to an S3 bucket the environment watches, and Airflow's scheduler and workers execute the tasks. Choose MWAA when the scenario describes complex dependency graphs, backfills over historical date ranges, a team that already writes Airflow DAGs, or orchestration that must reach beyond AWS through Airflow's large operator ecosystem (on-premises systems, other clouds, third-party tools). The trade-off is operational shape: a provisioned MWAA environment bills per environment-hour for as long as it exists — including the idle hours between nightly runs — so for a purely AWS-native pipeline with a handful of steps, Step Functions (which costs nothing while no execution runs) is almost always the cheaper, lower-overhead answer. Amazon MWAA Serverless (November 2025) removes that idle charge by billing for the time tasks actually run, which changes the comparison when a team wants Airflow DAGs for infrequent workloads; the always-on charge applies to provisioned environments.

A third, narrower option is AWS Glue workflows: chains of Glue crawlers and Glue jobs connected by triggers (scheduled, on-demand, conditional, or EventBridge-event based). If everything in the pipeline is a Glue crawler or Glue job, a Glue workflow keeps orchestration inside one service with nothing extra to build — but the moment the pipeline needs Lambda, Athena, EMR, or human-style branching, you have outgrown it.

Match the automation service to the workflow's complexity and trigger style — the exam signals the intended answer with phrases like least operational overhead, existing Airflow DAGs, or only Glue jobs and crawlers.

Two gotchas the exam likes. First, EventBridge schedules things; it does not orchestrate them — it can start a Glue job at midnight, but it cannot wait for the job, check the result, and start the next step. If the question involves sequencing with failure handling, the answer is Step Functions or MWAA, with EventBridge merely providing the kickoff. Second, do not chain Lambda functions by having one invoke the next: that hand-rolled orchestration loses retries, visibility, and state, and the exam-correct refactor is Step Functions.

Troubleshooting a managed workflow starts with the service's own execution record: Amazon MWAA writes each Airflow component's logs to its own Amazon CloudWatch log group, Step Functions keeps a per-execution event history (next section), and AWS Glue workflows expose the run state of each node.

MWAA log groups: which one answers which symptom

When logging is enabled for a component, MWAA publishes it to a log group named airflow-<environment>-<Component>:

A DAG that never appears, or appears and vanishes. First confirm the file is under the S3 DAGs prefix the environment watches; then read the DAGProcessing log for the import error. Airflow re-imports DAG files continuously, and an import that exceeds dagbag_import_timeout (30 seconds by default) fails. Top-level code — an API call or database query at module level to build the task list — therefore makes a DAG disappear intermittently whenever that call is slow. The fix is to move the work into tasks (or read a cached configuration file), not to raise the timeout.

Tasks that do not run: tell the two causes apart. Tasks stuck in queued while the worker count sits at its configured maximum means worker saturation. MWAA autoscales workers between a minimum (the floor kept while idle) and a maximum (the peak ceiling), so the fix is a higher maximum worker count or a larger environment class — the minimum is only the floor kept while idle, so raising it adds idle cost without lifting the peak ceiling. Tasks held in scheduled at a fixed ceiling while workers sit idle is different: an Airflow concurrency limit is capping them, such as max_active_tasks on the DAG or core.max_active_tasks_per_dag (Airflow default 16). Raise the limit through the DAG argument or an Airflow configuration option; more workers will not help.

Dependency failures. The environment references requirements.txt and plugins.zip by S3 object version, so uploading a new file to the bucket changes nothing until you update the environment to point at the new version. Packages are installed as the Airflow components start; when an install fails, read the requirements_install_<ip> streams in the scheduler log group. Pin versions deliberately (with the Airflow constraints file) and test changes on a staging environment or the MWAA local runner. An environment on private network access with no internet route cannot reach PyPI: package the wheels into plugins.zip and point pip at them in requirements.txt with --find-links /usr/local/airflow/plugins and --no-index.

Glue workflows stall when a conditional trigger's condition is never met — for example, a predecessor job failed, so the trigger that waits for its success never fires. The workflow run view shows which node stopped the chain; the job's own CloudWatch logs and error output explain why. A job rerun after fixing bad input may also need attention to its job bookmark: bookmarks track what was already processed, so reprocessing corrected data can require resetting or rewinding the bookmark.

Troubleshooting a Step Functions run means reading what happened to each state — and for a Standard workflow that record exists without any setup, because the service keeps every execution's event history (the input, output and error of every state), viewable in the console or through GetExecutionHistory.

Logging is a separate, optional layer. Sending execution events to CloudWatch Logs is off unless you configure a logging level (ALL, ERROR or FATAL) on the state machine. Standard workflows have their history either way; Express workflows do not keep one, so CloudWatch Logs is their only per-execution record.

Read the error name. States.TaskFailed wraps an error thrown by the invoked service; States.Timeout means a state ran longer than its TimeoutSeconds or missed its heartbeat; States.DataLimitExceeded means a state's input or output passed the 256 KiB payload limit, which is the same for Standard and Express. Raising a timeout or switching workflow type does not fix the last one: write the data to Amazon S3 and pass only the bucket and key between states (or let a Distributed Map read its items from S3 directly). If a downstream state received the wrong input, inspect the input/output processing (paths, selectors) in the history rather than the worker code.

Retry inside an execution versus rerunning a failed one. A state's Retry handles transient errors within one execution — a throttled call succeeds on the second attempt. It does nothing for an execution that has already failed: a fresh rerun starts again from the first state. Two mechanisms make reruns cheap:

• Redrive restarts a failed, aborted or timed-out Standard execution from the state that failed, within 14 days of the failure, keeping the results of the steps that succeeded. It reuses the original state machine definition and input, so only fixes made outside the definition take effect — for example, corrected Lambda code published behind the alias the state calls. A change to the definition itself needs a new execution.
• A persisted checkpoint — your own record of progress, such as the last cursor processed, kept in DynamoDB or S3 — lets a new execution resume where the last one stopped. This is the pattern for paged extracts (see Consuming and maintaining data APIs).

Pipeline code reaches AWS features through the SDKs — boto3 in Python being the one you will see most — and the exam tests the habits that make SDK calls production-safe rather than the syntax.

Credentials come from IAM roles, never from code. A Lambda function, Glue job, or EMR step gets temporary credentials from its execution role automatically; the SDK picks them up from the environment with no keys embedded anywhere.

Retries are built in, and they do not slow you down. The SDKs retry throttled and transient failures with exponential backoff, configured through a retry mode rather than hand-written loops. Standard mode retries with jittered backoff; adaptive mode adds client-side rate limiting (a token bucket that delays requests after throttling), and boto3 documents adaptive mode as experimental. The distinction matters when many threads hammer one API: retries alone do not lower the request rate, so when retries are exhausted the fix is to reduce concurrency, batch the calls, or use adaptive mode — not to add more attempts. SDK retry modes apply only to AWS service calls made through the SDK; calls to a third-party REST API with an HTTP library need their own backoff.

Partial success is not an exception. Some batch APIs succeed while leaving work undone. DynamoDB BatchWriteItem returns throttled writes in UnprocessedItems inside a successful response, so the SDK's retries never fire; the caller must resubmit those items with backoff, or use the higher-level Table.batch_writer(), which resends them for you.

Pagination is explicit. List operations return one page plus a continuation token. S3 ListObjectsV2 returns at most 1,000 keys per call and MaxKeys cannot raise that, so a script that lists a large prefix once silently sees only the first 1,000 objects. Use the SDK's paginators to iterate every page. Waiters poll a resource until it reaches a state (a cluster becomes available, a table becomes active) so you do not hand-roll sleep loops.

Many data-service APIs are asynchronous: StartJobRun (Glue), StartQueryExecution (Athena), and ExecuteStatement (Redshift Data API) return immediately with an identifier, and your code either polls a status call or — better — reacts to the completion event in EventBridge. Glue job parameters are passed in Arguments with keys prefixed by -- (for example --source_path) and read inside the script with getResolvedOptions(sys.argv, ['source_path']).

When a Lambda function must query Redshift without managing connections, use the Redshift Data API. It lets Lambda, Step Functions, and notebooks run SQL against Redshift over HTTPS with IAM or Secrets Manager authentication — no JDBC driver, no persistent connection, and no VPC networking from the caller's side. BatchExecuteStatement runs up to 40 SQL statements serially as a single transaction, so a stage-merge-cleanup sequence either commits as a whole or rolls back. Setting WithEvent publishes an event to EventBridge when the statement finishes, so the caller does not need to poll DescribeStatement.

The anti-pattern to recognize: a Lambda function that calls StartQueryExecution and then sleeps in a loop waiting for the result is burning paid compute on waiting. The better designs are a Step Functions state machine that waits natively (the run-a-job integration) or polls between states, or an EventBridge rule on the completion event.

Before adding orchestration code, use the automation features the processing services already ship — DEA-C01 rewards answers that let Glue, EMR, and Redshift do work natively instead of bolting on custom infrastructure.

AWS Glue automates incrementality with job bookmarks (each run processes only data added since the last successful run), chains work with triggers and workflows, scales with auto scaling worker fleets, and offers Flex execution for non-urgent jobs at lower cost. Glue streaming jobs run Spark Structured Streaming continuously against Kinesis or Kafka when the transformation itself must be always-on. Crawlers keep the Data Catalog's schemas and partitions synchronized as new data lands, which is itself automation — downstream Athena queries see new partitions without manual DDL.

Amazon EMR processes at cluster scale, and its automation primitive is the step: you submit Spark or Hive steps to a cluster through the API (AddJobFlowSteps, or at cluster creation). Steps run in order, one at a time at the default step concurrency of 1 (you can raise it for independent steps). Each step's ActionOnFailure decides what a failure does: CONTINUE moves on to the next step, CANCEL_AND_WAIT cancels the remaining steps and leaves the cluster running for investigation, and TERMINATE_CLUSTER shuts the cluster down. A transient cluster (auto-terminate after the last step) is the classic cost-controlled batch pattern, usually driven by Step Functions. An auto-termination policy is different: it shuts a cluster down only after it has been idle for the timeout you set — a safety net for long-running or interactive clusters rather than a per-job lifecycle. Managed scaling resizes long-running clusters with load, and EMR Serverless removes cluster management entirely: you submit jobs to an application and pay for the resources the job actually uses, the right answer when the scenario wants Spark without capacity planning.

Amazon Redshift automates transformation inside the warehouse: stored procedures encapsulate multi-statement SQL transformations, scheduled queries run them on a cron (driven by EventBridge and the Data API under the hood), materialized views with auto refresh keep precomputed aggregations current, and auto-copy from S3 loads new files as they arrive without an external trigger. Auto-copy needs an S3 event integration between the bucket and the warehouse, is defined with COPY ... JOB CREATE <name> AUTO ON, and records which files each job has loaded so no file loads twice — whereas a plain COPY run on a schedule reloads every file under the prefix unless you move or track the files yourself. When a question describes recurring SQL-shaped transformation on data already in Redshift, the ELT answer — run it in Redshift on a schedule — beats exporting the data to an external processing service and loading it back.

Preparing data for transformation means profiling it, fixing quality issues, and standardizing formats before the heavy processing runs — and the two services the exam names for this are AWS Glue DataBrew and Amazon SageMaker Unified Studio.

DataBrew is visual, no-code data preparation. You open a project on a sample of a dataset (from S3, the Glue Data Catalog, Redshift, or JDBC sources), build a recipe from its library of more than 250 built-in transformations — remove or fill nulls, standardize casing and date formats, split and merge columns, deduplicate, handle outliers — and then run the recipe as a recipe job against the full dataset on a schedule, writing cleaned output to S3. Profile jobs generate statistics about a dataset (completeness, distributions, correlations, duplicates) — by default from a sample of the first 20,000 rows, so set the job's sample to the full dataset when you need whole-dataset statistics — and data quality rules validate datasets against thresholds you define. The exam signal for DataBrew is a scenario where analysts or less-technical users must clean and normalize data without writing code; if the transformation needs custom logic or joins at scale, a Glue Spark job is the better fit.

SageMaker Unified Studio is the single environment that brings AWS analytics and machine learning tools together: within a governed project, teams share access to data through the lakehouse architecture, prepare it with visual ETL flows or JupyterLab notebooks, query it with built-in SQL editors, and publish assets to a business catalog for discovery. Sharing across teams is governed through that catalog: the producing project publishes a prepared asset, a consuming project subscribes to it, and once the owner approves the subscription, access is granted — the governed alternative to bucket policies or copying data between teams. For Task 3.1, its relevance is collaborative preparation: instead of each engineer stitching together consoles, a project gives data engineers, analysts, and scientists one governed place where prepared datasets, queries, and notebooks live together. When a scenario emphasizes a unified, project-based environment spanning data preparation through analytics and ML, Unified Studio is the answer.

Positioning matters on the exam: DataBrew is the no-code cleanup tool, Glue ETL is the code-first Spark engine, and Unified Studio is the umbrella workspace where preparation happens alongside querying and model work — they complement rather than replace each other.

Amazon Athena is the serverless query layer of an automated pipeline: it runs standard SQL directly against data in S3 using Glue Data Catalog schemas, with no infrastructure and pricing based on data scanned. Inside pipelines it appears two ways. First, as an automated step: Step Functions can run an Athena query natively, and code calls StartQueryExecution and reacts to completion — used for validation queries, report extracts, or transformations expressed as CTAS (CREATE TABLE AS SELECT), which materializes query results back to S3 in a columnar format like Parquet, partitioned and compressed. Second, as the verification tool: after a load, an Athena query that counts rows or checks for nulls is the cheapest automated quality gate. Cost and performance discipline carries the marks: partition the data and use columnar formats so queries scan less, and use workgroups to separate teams and control spend.

Workgroups have two kinds of data usage control, and they behave differently: a per-query control cancels any query that scans more than its limit, while workgroup-wide controls track total data scanned over a period (such as an hour or a day) and only trigger an alarm action such as an SNS notification — they do not stop queries. To stop a single runaway scan, use the per-query limit. Turning on override client-side settings forces every query in the workgroup to use the workgroup's result location and encryption, whatever the client requests.

A data API is consumed with authentication, pagination, polite backoff and rerun-safe writes, and maintained with per-client throttling, caching and versioning that never breaks existing callers.

Consuming a data API means calling someone else's interface politely. Authenticate as required (API keys, IAM SigV4, or OAuth tokens) and page through results with the API's continuation tokens. When the API throttles with HTTP 429, honour its Retry-After header and back off exponentially with jitter: when many concurrent callers are throttled together, backoff without jitter — or every caller waiting exactly the Retry-After value — sends them back in lockstep and they collide again. SDK retry modes do not help here; they cover AWS service calls only, not a third-party API called with an HTTP library.

Reruns are inevitable, and a safe one needs two properties. Idempotent: each page is written to a deterministic S3 key derived from the request (such as the extract date and cursor), so a rerun overwrites rather than duplicates. Resumable: the next-page cursor is checkpointed (for example in DynamoDB) after each page lands, so a rerun continues from the last good page instead of the first. An extract that can outlast Lambda's 15 minutes is typically a Step Functions loop that calls a Lambda function once per page. A Lambda function on an EventBridge schedule that pulls an API and writes raw responses to S3 is the canonical small-scale pattern.

Maintaining a data API is the other direction: exposing your data to other systems, typically with Amazon API Gateway in front of Lambda, which queries the backing store (DynamoDB for key-value lookups, Athena or Redshift for analytical reads). Maintenance is what the exam probes:

• Per-client limits: usage plans tie each consumer's API key to its own rate, burst and quota. They are a feature of API Gateway REST APIs; HTTP APIs have no usage plans or API keys (their throttling is set per stage and route), and Lambda function URLs and Application Load Balancers have neither.
• Caching at the API layer (a REST API stage cache) absorbs repeated reads.
• Versioning: publish a breaking change as a new version — a /v2 path, stage, or base-path mapping on the custom domain — while v1 keeps serving existing consumers until they migrate. The principle is that old and new contracts must be addressable side by side: any change that routes existing callers to the new contract — whether all at once or a percentage at a time, as a canary release does while testing a deployment — breaks consumers who have not migrated.
• Monitoring of latency and error rates tells you when the backend store no longer fits the access pattern.

Amazon EventBridge supplies the triggers for automated pipelines in three forms: rules that react to events on a bus, EventBridge Scheduler for time-based invocations, and EventBridge Pipes for point-to-point integration from a stream or queue.

Rules pattern-match events on an event bus and route them to targets: S3 object-created events (delivered via EventBridge) can start a Step Functions execution with filtering on bucket and key; a Glue job state change event with state FAILED can route to an SNS topic or a remediation Lambda — the standard way to react to pipeline failures without polling. A rule can also run on a schedule, but scheduled rules evaluate cron expressions in UTC only.

EventBridge Scheduler is the newer, richer way to run things on time, and through universal targets it can invoke nearly any AWS API action directly as its target:

• Cron and rate schedules in a chosen time zone (daylight saving handled), and one-time schedules. Setting ActionAfterCompletion to DELETE removes a schedule after its last invocation; it does not react to failures.
• A flexible time window spreads invocations across a window to smooth load on the target; it controls when a run starts, not what happens when it fails.
• A retry policy (maximum retry attempts and maximum event age) retries an invocation that fails with a retryable error — throttling or a server-side error from the target API. Non-retryable errors, such as missing permissions, are not retried. An SQS dead-letter queue captures invocations that still failed, so nothing is lost silently.
• Scheduler retries the API call it makes. A Glue job's own MaxRetries is a different layer: it re-runs a job run that started and then failed, not a StartJobRun request that was rejected.

EventBridge Pipes connects one source to one target, with optional filtering and enrichment (a Lambda function, Step Functions Express workflow, API Gateway or API destination) in between — no code to poll the source. It is how a DynamoDB stream, Kinesis stream, SQS queue or Amazon MSK topic reaches a target such as Step Functions, because DynamoDB stream records never appear on an event bus, so a rule cannot match them.

Archive and replay give event-driven pipelines a recovery path. An archive stores events from a bus — only those received after the archive was created, optionally filtered by a pattern. A replay sends archived events from a chosen time range back to the source bus, and can be limited to specific rules, so after fixing a buggy consumer you can replay only to that consumer's rule.

AWS Lambda is the automation glue of Task 3.1: it runs code in response to events with no servers, which makes it the default executor for lightweight processing — validating and tagging a file the moment it lands in S3, transforming small payloads, invoking service APIs, and dispatching notifications. Its boundaries define its exam role: execution time is capped at 15 minutes and memory at 10,240 MB (CPU scales with memory), so Lambda is the wrong answer for heavy transformation of large datasets — that work belongs in Glue or EMR, with Lambda merely starting it. Concurrency is the other lever: hundreds of S3 events can invoke hundreds of parallel executions, which is powerful for fan-out but demands idempotent handlers and awareness of downstream limits (a database receiving hundreds of simultaneous connections, for instance).

Event source mappings are a Lambda-only feature: an event source mapping is the poller that reads a DynamoDB stream, Kinesis stream or SQS queue and invokes a function with batches of records. Other targets reach those sources through EventBridge Pipes (previous section).

Scenario

A retail company receives supplier inventory files in S3 throughout the night and needs them validated, transformed to Parquet, made queryable, and reported on by 7 AM — with alerts on any failure and no idle infrastructure. The exam-shaped design: S3 events (via EventBridge) trigger a Step Functions Standard workflow per file batch; the state machine runs a Lambda validation step, then a Glue job (run-a-job pattern, so the workflow waits) to convert and partition the data, then a Glue crawler to register new partitions, then an Athena CTAS query for the morning aggregate. Every state has Retry with backoff; a Catch route publishes failures to SNS. An EventBridge Scheduler cron fires a final 6 AM completeness check. Nothing runs — or bills — between files, and every failure is visible in one execution history.