Transform and Process Data on AWS: Glue, EMR, Lambda, and Redshift

Match the engine to the workload's volume (how much data), velocity (how fast it arrives and how fresh results must be), and variety (structured tables, semi-structured JSON, unstructured text and images) — the three Vs the exam expects you to define and apply. Structured data has a fixed, predefined schema (relational rows with typed columns); semi-structured data is self-describing through field names or tags but its fields vary between records (JSON, XML, Avro); unstructured data has no inherent fields at all (scanned documents, images, audio, video) until a service extracts them. Volume decides single-function versus distributed processing, velocity decides batch versus streaming, and variety decides whether you parse (structured and semi-structured) or must first extract meaning (unstructured, covered in the last section).

The exam's decision signals: "file arrives, transform it, under 15 minutes" is Lambda; "serverless ETL with the least management" is Glue; "existing Spark/Hadoop jobs, petabyte scale, or a specific framework" is EMR; "analysts who know SQL, data already loaded" is Redshift ELT. When two answers both work, pick the one with less to operate at the stated scale.

Lambda's limits are hard limits. A function runs for at most 15 minutes with at most 10,240 MB of memory, so a file that fails at exactly 15 minutes at the largest memory setting can never finish in Lambda — route the large files to a Glue Spark job and keep Lambda for the small, frequent ones. More memory, asynchronous invocation or provisioned concurrency do not change the ceiling. For a Lambda function reading a Kinesis data stream, the event source mapping processes one batch per shard at a time by default; when iterator age rises because each invocation waits on I/O, raise the parallelization factor (up to 10 concurrent batches per shard) — Lambda still keeps records with the same partition key in order, so ordering survives without adding shards.

Redshift ELT keeps the data where it is. When the data is already loaded, transform it with SQL inside the warehouse — stored procedures, CREATE TABLE AS, and materialized views (with incremental refresh) — instead of unloading it to S3, transforming in Spark, and reloading it. Exporting adds a pipeline, two copies, and data movement for work the warehouse already does.

AWS Glue runs serverless Apache Spark: you choose a worker type and count and pay per DPU-hour while the job runs. Glue offers three job types — Spark (batch ETL), Spark streaming (continuous ETL from Kinesis or Kafka) and Python shell (plain Python on a fraction of a DPU). AWS Glue for Ray closed to new customers on 30 April 2026, so do not plan new work on it. Glue has no Hive engine: Hive scripts must be rewritten for Glue, or run unchanged on Amazon EMR (including EMR Serverless). Capacity levers are the worker type — G.1X for general workloads, G.2X and larger (up to G.16X) for heavy joins and aggregations, memory-optimized R workers (R.1X–R.8X) for jobs that keep running out of memory — the number of workers, and auto scaling, which releases idle executors mid-job so you stop paying for them.

Job bookmarks: incremental processing

Job bookmarks persist state from the last successful run so the next run processes only new data — the difference between an incremental daily load and re-reading the whole lake. They only work when the script is wired for them:

• Bookmarks must be enabled on the job (job-bookmark-enable).
• The script calls job.init() at the start and job.commit() at the end; without the commit, the run's state is never saved and the next run re-reads everything.
• Each source is read through a Glue DynamicFrame API (for example create_dynamic_frame.from_catalog) with a transformation_ctx name — the key under which state is stored. Data read with plain spark.read is not tracked.

The Pause option (job-bookmark-pause) reads from the last saved state but never updates it, so a job left paused keeps re-reading the same files even though it commits; Disable ignores state and processes everything. Auto scaling, more workers or Flex change how fast or cheaply a job runs, never what it reads — none of them makes a job incremental.

Partition pruning: pushdown predicates

If the lake is partitioned by date, a push_down_predicate makes Glue read only matching partitions instead of loading everything and filtering in memory with filter() or a DataFrame where — which still reads every file. The predicate is applied after Glue lists all of the table's partitions from the Data Catalog, though, so on a table with millions of partitions the listing itself becomes the slow step. Add a partition index on the table and use catalogPartitionPredicate: the Data Catalog then filters partitions on the server side and returns only the matches.

DynamicFrames and their transforms

Glue's DynamicFrame extends the Spark DataFrame for messy, semi-structured data: it tolerates a different schema per record and records a choice type when a field's type varies (for example price arriving as both string and number). Each built-in transform fixes one specific problem:

Convert to a standard DataFrame (toDF()) for full Spark SQL, and back with DynamicFrame.fromDF when you need Glue sinks or DynamicFrame transforms (bookmarks only track sources read through DynamicFrame APIs, so fromDF adds no bookmark tracking).

Glue Flex execution runs non-urgent Spark jobs on spare capacity at a lower DPU rate (Flex supports only G.1X and G.2X workers) — the answer when a nightly job has a loose SLA and the question asks for cheaper. For jobs that are not Spark-shaped at all — a small API pull, a file rename sweep — a Python shell job is far cheaper than spinning up Spark executors. The Glue Data Catalog underpins all of it: crawlers or explicit DDL register tables and partitions once, and Glue, Athena, EMR and Redshift Spectrum all query the same metadata.

EMR gives you the full open-source big-data stack — Spark, Hive, Flink, Presto/Trino, HBase — with three deployment models the exam distinguishes.

On EMR on EC2, put the primary node and core nodes (which hold HDFS) on On-Demand or reserved capacity and run task nodes on Spot Instances, since a reclaimed task node loses only compute, not data. Instance fleets let you list several instance types per node group and use the price-capacity-optimized allocation strategy, so EMR launches Spot capacity from the pools with the most spare capacity at a low price — fewer interruptions than pinning one instance type. For a cluster that must stay up all day while load swings, EMR managed scaling resizes it automatically from workload metrics within the minimum and maximum you set; custom automatic scaling policies do the same only with rules you write and tune yourself. EMR on EKS maps a virtual cluster to a Kubernetes namespace (one per team is common), and each job run chooses its own EMR release, so teams can pin different Spark versions on the same EKS cluster.

Neighbouring services are not substitutes. Amazon Managed Service for Apache Flink runs Flink applications only — it cannot host HBase or node-level software installs. AWS Glue runs Spark and Python, not Hive. When a requirement lists HBase, custom node software, or several frameworks on one cluster, EMR on EC2 is the fit.

Optimizing container usage on EKS and ECS

Optimizing container usage on Amazon EKS and Amazon ECS is its own skill statement. The levers: right-size task CPU and memory from observed utilization rather than guesses — over-provisioned requests strand capacity, under-provisioned ones cause OOM kills; bin-pack containers onto fewer nodes to raise utilization; use Spot capacity for interruption-tolerant batch processing and executors; and choose Fargate when you want no node management at all versus EC2-backed capacity when sustained utilization is high enough that managing nodes pays for itself. Horizontal scaling policies (and on EKS, an autoscaler such as Karpenter) keep the fleet matched to queue depth. A scheduled container that should run with no servers to patch is an ECS task on Fargate started by EventBridge Scheduler.

Fargate has no GPUs. A GPU-accelerated step (an OCR or inference model, for example) needs EC2 capacity with GPU instances — on ECS, a capacity provider backed by a GPU Auto Scaling group alongside the Fargate tasks in the same cluster; on EKS, a GPU node group. Asking for bigger Fargate sizes adds CPU and memory, never a GPU, and Lambda has no GPU option either.

Decision rule: containers on ECS/EKS suit custom transformation code and frameworks that are not Spark-shaped, or organizations standardized on Kubernetes; if the workload is standard Spark ETL and no such standardization exists, Glue or EMR Serverless is less to operate.

Transformation engines reach relational sources through JDBC (Java-based tools — Glue, Spark, EMR) and ODBC (BI tools and other clients); Redshift exposes both driver types for anything that speaks SQL to it. In Glue, a connection object holds the JDBC URL, a reference to the credentials, and the VPC subnet and security group the job uses to reach private databases (networking and credentials are covered in the next section). When reading over JDBC, parallelize by partitioning the read on an evenly distributed column (hashfield for any column or hashexpression for a whole-number column, with hashpartitions, in Glue; or Spark's partition column and bounds) and push filters into the source query so you do not drag the whole table across the network.

Integrating data from multiple sources has several patterns, ordered by how much data actually moves:

• Query in place (federated query): Amazon Athena federated query uses Lambda-based data source connectors to join S3 data with DynamoDB, RDS and other sources in one SQL statement. A Glue crawler can catalogue a DynamoDB table, but Athena still cannot query it through that catalog entry — it needs the Athena DynamoDB connector. Amazon Redshift federated queries reach into Aurora and RDS PostgreSQL/MySQL live on every query: nothing is stored in Redshift and every query loads the source. Redshift Spectrum reads only Amazon S3, never operational databases.
• Zero-ETL integrations: Aurora MySQL, Aurora PostgreSQL and other supported sources replicate continuously into Redshift with no pipeline code, so Redshift stores a replicated copy. That is the fix when frequent Redshift federated queries are overloading the production database, and the least-overhead answer for keeping transactional data queryable in the warehouse.
• Physical consolidation: Glue or EMR jobs read each source, conform schemas, and join into the lake or warehouse — necessary when transformations are heavy or sources are numerous, at the cost of building and operating the pipeline.

The signals: "without moving the data" or "ad hoc, occasional" points to federated query; "stored in Redshift, continuously current, no load on the source and no ETL to manage" points to zero-ETL; recurring heavy joins across many sources point to a Glue/EMR pipeline into S3.

A Glue JDBC connection reaches a private database by placing elastic network interfaces (ENIs) in the connection's subnet, and those ENIs get private IP addresses only. That single fact explains most connection failures.

• Reaching Amazon S3. Inside the VPC, the job still needs S3 for scripts, temporary files and data. Give the subnet an S3 VPC endpoint or a route to a NAT gateway; a route to an internet gateway does not help, because the ENIs have no public IP. Glue checks for this when the run starts and fails with an endpoint/NAT validation error if it is missing. An S3 gateway endpoint serves only the route tables it is associated with — if the connection's subnet uses a custom route table and the endpoint is attached only to the main table, the subnet is not served.
• Security group. Glue's workers talk to each other through the same ENIs, so the connection's security group needs a self-referencing inbound rule allowing all TCP ports, plus outbound rules that allow that traffic (allow-all outbound is the simple case). Glue validates the security group at run start and fails with a security group validation error otherwise. The database's own security group must also allow the database port from the connection's group.

Credentials: Secrets Manager versus the alternatives

If the secret is encrypted with a customer managed AWS KMS key, the Glue job role needs kms:Decrypt on that key as well as secretsmanager:GetSecretValue — a missing key permission fails the job even when the secret policy is correct.

Convert row-based files (CSV, JSON) to Apache Parquet because analytics engines then read only the columns and row groups a query needs: Parquet is columnar, compresses far better than text (typically with Snappy), embeds the schema with proper types, and carries min/max statistics that let engines skip data entirely (predicate pushdown). Since Amazon Athena and Redshift Spectrum bill by bytes scanned, converting the lake to compressed Parquet routinely cuts both query cost and runtime by an order of magnitude. ORC is the comparable columnar alternative; Avro is row-based and better suited to record-at-a-time streaming than analytics.

Know what Parquet does not do. A SELECT * reads every column, so it gets no column-projection benefit; and row-group min/max statistics skip little when the filter column is unsorted, because almost every row group spans the whole range of values. What removes whole files from a scan on a frequently filtered column is partitioning on it (or sorting/clustering the data by it).

Three conversion paths to know:

• AWS Glue job — read CSV via the Data Catalog, write Parquet, usually adding partitioning (for example by date) in the same pass. The standard answer for recurring, scheduled conversion.
• Amazon Data Firehose record format conversion — converts incoming records to Parquet or ORC in flight using a Glue table for the schema, so streaming data lands in S3 already columnar. It reads JSON only (through the OpenX JSON SerDe or the Hive JSON SerDe); CSV or other input needs a Firehose Lambda transformation that turns each record into JSON first.
• Athena CTAS (CREATE TABLE AS SELECT) — one SQL statement that reads existing files and writes Parquet, ideal for one-off or SQL-team-driven conversions. By contrast, CREATE EXTERNAL TABLE … STORED AS PARQUET only declares a format for a location; it never rewrites existing CSV files, and pointing it at CSV data simply produces read errors.

Pair conversion with layout hygiene: partition on the columns queries filter by, and target file sizes of roughly 128 MB–1 GB, compacting the many small files that streaming delivery produces.

Scenario

A team lands about 500 GB of gzipped CSV daily and analysts query it in Athena, complaining of cost and slowness. A nightly Glue job (or Athena CTAS) that writes Snappy-compressed Parquet partitioned by event_date means typical queries scan a few gigabytes of relevant columns and partitions instead of the whole day's raw text — the canonical DEA-C01 "reduce Athena cost" answer.

Processing cost falls three ways: pay less per unit of compute, use fewer units, and scan less data — and exam answers usually combine them.

Pay less per unit. Run interruption-tolerant EMR task nodes (and ECS/EKS batch containers) on Spot Instances; run loose-SLA Glue jobs on Flex execution; choose EMR Serverless or Glue over an always-on EMR cluster when jobs are intermittent, because idle clusters bill around the clock. A persistent cluster only wins when utilization is sustained — then reserved capacity or Savings Plans apply. Note the scope: Compute Savings Plans cover EC2, Fargate and Lambda usage, not AWS Glue DPU-hours, so for Glue the per-unit levers are Flex, worker sizing and job type.

Use fewer units. Right-size Glue workers (a memory-bound job on more small workers can cost more than fewer large ones), enable Glue auto scaling so idle executors release mid-job, use EMR managed scaling and instance fleets with diversified Spot types, and make jobs incremental — bookmarks and CDC mean each run touches only new data. For sub-Spark tasks, a Glue Python shell job or Lambda costs a fraction of a Spark job's startup.

Scan less data. Columnar formats, compression, partition pruning, and pushdown predicates shrink what every downstream job and query reads — the format-conversion work above is itself a cost optimization. Compact small files so jobs stop paying per-file overhead, and tier aged raw data to cheaper S3 storage classes with lifecycle policies.

Two traps to recognize. First, an EMR cluster left running between nightly jobs is the classic "why is this expensive" scenario — the fix is transient clusters that terminate on completion, or EMR Serverless. Second, Spot for EMR core nodes risks losing HDFS blocks mid-job; the exam-safe pattern keeps core nodes On-Demand and puts Spot on task nodes only.

Diagnose transformation failures by matching the symptom to one of four recurring causes: memory exhaustion, data skew, small files, or source-side bottlenecks. Your evidence is CloudWatch job metrics and logs, and for Glue and EMR, the Spark UI — its stage and task views show exactly where time and memory go.

Executor out-of-memory (lost executors, containers killed) means partitions are too large for the worker: move to a larger worker (a bigger Glue G type, or a memory-optimized R type), increase shuffle partitions so each task handles less data, and stop caching what you do not reuse. Driver OOM is different — the driver ran out of memory, not the executors. The usual causes are pulling a dataset onto the driver with collect() (or toPandas()) and broadcast joins of large tables: Spark builds the broadcast table on the driver before shipping it to executors, so a broadcast hint on a multi-gigabyte table fails the driver however many workers you add. Remove the hint (let a shuffle join run) and keep work distributed.

Data skew shows up as stragglers: 199 tasks finish in a minute, one runs for an hour, because one join or group-by key (a null key, a dominant customer) holds a huge share of rows. Fixes: salt the hot key (append a random suffix to spread it across partitions, adjusting the join accordingly), broadcast the smaller side of the join so no shuffle of the big table is needed, or filter/handle the dominant key separately. Adding more executors does not fix skew — one task still holds the hot partition; that distinction is tested.

Two Spark defaults matter here. Adaptive Query Execution (AQE), including its automatic skew-join handling, is enabled by default from Glue 4.0, so on current Glue versions skew that persists usually needs an explicit fix. And Spark broadcasts a join side automatically only when it is under spark.sql.autoBroadcastJoinThreshold (10 MB by default); a reference table of, say, 200 MB is shuffled unless you add a broadcast() hint — the low-code skew fix when the other side is small enough to fit in memory.

Small files — thousands of KB-scale objects from streaming delivery — make Spark spend more time on per-file scheduling and S3 requests than on data. Compact them with a periodic Glue compaction job toward 128 MB–1 GB files. When the files cannot be rewritten, have Glue group them at read time with groupFiles (inPartition) and a target groupSize, so each task reads a batch of files. Glue turns grouping on automatically only when a read has more than 50,000 input files — below that you must set it yourself.

Also on the checklist: a JDBC source read serially becomes the whole job's bottleneck (partition the read); Lambda transforms failing at exactly 15 minutes have outgrown Lambda; a Kinesis-triggered Lambda falling behind needs a higher parallelization factor or more shards; and slow Redshift ELT is often WLM queue waiting or disk-spilling queries, visible in the system views, not raw cluster undersizing.

You expose transformed data to other systems through APIs rather than shared database credentials. For the warehouse, the Amazon Redshift Data API runs SQL over HTTPS without persistent JDBC connections, authenticates through IAM (or a Secrets Manager secret), and executes asynchronously: ExecuteStatement returns a statement ID, and you fetch status and results by that ID later. That connectionless model is what Lambda functions, Step Functions states and event-driven apps need, because they cannot efficiently hold connection pools. (Amazon RDS Proxy is not an alternative: it pools connections for RDS and Aurora databases and does not support Redshift.)

• Waiting is capped. WaitTimeSeconds lets ExecuteStatement wait for completion before responding, but only up to 30 seconds; a longer query just returns its current status.
• Completion as an event. Set WithEvent to true and the Data API sends an event to Amazon EventBridge when the statement finishes, so an EventBridge rule can start the next step with nothing polling.
• Long queries behind an API. API Gateway ends a request when its integration exceeds the timeout (29 seconds by default), so a multi-minute report cannot be returned synchronously however long the Lambda timeout is. Use two endpoints: one submits the query through the Data API and returns its ID; another fetches results by ID.

For general-purpose serving, front AWS Lambda with Amazon API Gateway: Lambda queries the store (DynamoDB for key-value lookups at scale, Athena or Redshift for analytics) and API Gateway adds authentication, throttling and caching. Choose the API type by the controls you need:

When data refreshes hourly and most callers ask for the same reports, stage caching with a matching TTL removes repeat load; when one client's bursts slow everyone, per-client usage-plan throttling contains it. Design paginated responses for large result sets rather than streaming unbounded payloads through a synchronous API.

Unstructured data must have its fields extracted before it can be transformed, and the right service depends on the input type:

Prefer the single managed feature that already does the job: for call recordings that need transcripts plus sentiment, post-call analytics is one job, whereas chaining standard Transcribe into Comprehend or Bedrock adds a stage and code.

Large language models enter data engineering as a processing step for unstructured and messy data, invoked through Amazon Bedrock's API with no model training and no servers: extracting structured fields from free text (support tickets, contracts), classifying and tagging records, summarizing long text into analytics-friendly fields, generating embeddings for semantic search, and assisting entity resolution. Treat the model as one transformation stage, and pick the invocation option by the workload:

Engineering cautions still apply: per-call cost and rate limits make batching and caching important, retries need backoff, and non-deterministic output should be validated against a schema before loading. The signal is unstructured data that needs understanding, not just parsing: regex and string functions handle formats; LLMs handle meaning.