Transform and Process Data on AWS: Glue, EMR, Lambda, and Redshift

Match the engine to the workload's volume (how much data), velocity (how fast it arrives and how fresh results must be), and variety (structured tables, semi-structured JSON, unstructured text and images) — the three Vs the exam expects you to define and apply. Structured data has a fixed schema (relational rows); semi-structured carries its own flexible schema (JSON, Avro); unstructured has none (documents, images, audio).

The exam's decision signals: "file arrives, transform it, under 15 minutes" is Lambda; "serverless ETL with the least management" is Glue; "existing Spark/Hadoop jobs, petabyte scale, or a specific framework" is EMR; "analysts who know SQL, data already loaded" is Redshift ELT. When two answers both work, pick the one with less to operate at the stated scale.

Glue runs Apache Spark without clusters: you choose a worker type and count, and pay per DPU-hour while the job runs. Capacity levers are the worker type — G.1X for general workloads, G.2X and larger (up to G.8X) for memory-heavy joins and aggregations — the number of workers, and auto scaling, which releases idle executors mid-job so you stop paying for them.

Three features carry the most exam weight. Job bookmarks persist the position of the last successful run so the next run processes only new data — the difference between an incremental daily load and re-reading the whole lake. Pushdown predicates and partition pruning filter at the source: if the lake is partitioned by date, a pushdown predicate makes Spark list and read only the matching partitions instead of scanning everything then filtering in memory. Glue Flex execution runs non-urgent jobs on spare capacity at a lower DPU rate — the answer when a nightly job has a loose SLA and the question asks for cheaper.

Glue's DynamicFrame extends the Spark DataFrame for messy data: it tolerates inconsistent schemas per record, and ResolveChoice handles fields whose type varies across rows — useful when ingesting semi-structured JSON. You can convert to a standard DataFrame (toDF()) for full Spark SQL, and back.

For jobs that are not Spark-shaped at all — a small API pull, a file rename sweep — a Glue Python shell job runs plain Python on a fraction of a DPU, far cheaper than spinning up Spark executors. The Glue Data Catalog underpins all of it: crawlers or explicit DDL register tables and partitions once, and Glue, Athena, EMR, and Redshift Spectrum all query the same metadata.

EMR gives you the full open-source big-data stack — Spark, Hive, Flink, Presto/Trino, HBase — with three deployment models the exam distinguishes. EMR on EC2 offers maximum control and the lowest cost at sustained scale: put core nodes (which hold HDFS) on On-Demand or reserved capacity and run task nodes on Spot Instances, since a reclaimed task node loses only compute, not data. Instance fleets diversify across instance types to reduce Spot interruptions, and managed scaling resizes the cluster with load. EMR Serverless submits Spark or Hive jobs with no cluster at all — pay per vCPU- and memory-second — the least-overhead EMR answer for intermittent workloads. EMR on EKS runs Spark jobs as containers on an existing Amazon EKS cluster, so data workloads share capacity (and idle headroom) with the rest of the organization's Kubernetes services instead of running a dedicated cluster.

Optimizing container usage on Amazon EKS and Amazon ECS is its own skill statement. The levers: right-size task CPU and memory from observed utilization rather than guesses — over-provisioned requests strand capacity, under-provisioned ones cause OOM kills; bin-pack containers onto fewer nodes to raise utilization; use Spot capacity for interruption-tolerant batch processing; and choose Fargate when you want no node management at all versus EC2-backed capacity when sustained utilization is high enough that managing nodes pays for itself. Horizontal scaling policies (and on EKS, an autoscaler such as Karpenter) keep the fleet matched to queue depth.

Decision rule for the exam: containers on ECS/EKS suit custom transformation code and frameworks that are not Spark-shaped, or organizations standardized on Kubernetes; if the workload is standard Spark ETL and no such standardization exists, Glue or EMR Serverless is less to operate.

Transformation engines reach relational sources through JDBC (Java-based tools — Glue, Spark, EMR) and ODBC (BI tools and other clients); Redshift exposes both driver types for anything that speaks SQL to it. In Glue, a connection object stores the JDBC URL and credentials and — critically — attaches elastic network interfaces inside your VPC subnets so the job can reach private databases; store the credentials in AWS Secrets Manager and reference the secret rather than embedding passwords in job parameters. When reading over JDBC, parallelize by partitioning the read on a numeric column and push filters down into the source query so you do not drag the whole table across the network.

Integrating data from multiple sources has several exam-relevant patterns, ordered by how much data actually moves:

• Query in place: Amazon Athena federated query uses Lambda-based connectors to join S3 data with DynamoDB, RDS, or other sources in one SQL statement; Amazon Redshift federated queries reach into Aurora and RDS PostgreSQL/MySQL live. Best for occasional cross-source analysis without building a pipeline.
• Zero-ETL integrations: Aurora (and other supported sources) replicate continuously into Redshift with no pipeline code — the least-overhead answer for keeping transactional data queryable in the warehouse.
• Physical consolidation: Glue or EMR jobs read each source, conform schemas, and join into the lake or warehouse — necessary when transformations are heavy or sources are numerous, at the cost of building and operating the pipeline.

The exam signal: "without moving the data" or "ad hoc, occasional" points to federated query; "continuously available in Redshift with no ETL to manage" points to zero-ETL; recurring heavy joins across many sources point to a Glue/EMR pipeline into S3.

Convert row-based files (CSV, JSON) to Apache Parquet because analytics engines then read only the columns and row groups a query needs: Parquet is columnar, compresses far better than text (typically with Snappy), embeds the schema with proper types, and carries min/max statistics that let engines skip data entirely (predicate pushdown). Since Amazon Athena and Redshift Spectrum bill by bytes scanned, converting the lake to compressed Parquet routinely cuts both query cost and runtime by an order of magnitude. ORC is the comparable columnar alternative; Avro is row-based and better suited to record-at-a-time streaming than analytics.

Three conversion paths to know:

• AWS Glue job — read CSV via the Data Catalog, write Parquet, usually adding partitioning (for example by date) in the same pass. The standard answer for recurring, scheduled conversion.
• Amazon Data Firehose record format conversion — converts incoming JSON to Parquet or ORC in flight using a Glue table for the schema, so streaming data lands in S3 already columnar with zero transformation code.
• Athena CTAS (CREATE TABLE AS SELECT) — one SQL statement that reads existing files and writes Parquet, ideal for one-off or SQL-team-driven conversions.

Pair conversion with layout hygiene: partition on the columns queries filter by, and target file sizes of roughly 128 MB–1 GB, compacting the many small files that streaming delivery produces.

Scenario

A team lands about 500 GB of gzipped CSV daily and analysts query it in Athena, complaining of cost and slowness. A nightly Glue job (or Athena CTAS) that writes Snappy-compressed Parquet partitioned by event_date means typical queries scan a few gigabytes of relevant columns and partitions instead of the whole day's raw text — the canonical DEA-C01 "reduce Athena cost" answer.

Processing cost falls three ways: pay less per unit of compute, use fewer units, and scan less data — and exam answers usually combine them.

Pay less per unit. Run interruption-tolerant EMR task nodes (and ECS/EKS batch containers) on Spot Instances; run loose-SLA Glue jobs on Flex execution; choose EMR Serverless or Glue over an always-on EMR cluster when jobs are intermittent, because idle clusters bill around the clock. A persistent cluster only wins when utilization is sustained — then reserved capacity or Savings Plans apply.

Use fewer units. Right-size Glue workers (a memory-bound job on more small workers can cost more than fewer large ones), enable Glue auto scaling so idle executors release mid-job, use EMR managed scaling, and make jobs incremental — bookmarks and CDC mean each run touches only new data. For sub-Spark tasks, a Glue Python shell job or Lambda costs a fraction of a Spark job's startup.

Scan less data. Columnar formats, compression, partition pruning, and pushdown predicates shrink what every downstream job and query reads — the format-conversion work above is itself a cost optimization. Compact small files so jobs stop paying per-file overhead, and tier aged raw data to cheaper S3 storage classes with lifecycle policies.

Two traps to recognize. First, an EMR cluster left running between nightly jobs is the classic "why is this expensive" scenario — the fix is transient clusters that terminate on completion, or EMR Serverless. Second, Spot for EMR core nodes risks losing HDFS blocks mid-job; the exam-safe pattern keeps core nodes On-Demand and puts Spot on task nodes only.

Diagnose transformation failures by matching the symptom to one of four recurring causes: memory exhaustion, data skew, small files, or source-side bottlenecks. Your evidence is CloudWatch job metrics and logs, and for Glue and EMR, the Spark UI — its stage and task views show exactly where time and memory go.

Executor out-of-memory (lost executors, containers killed) means partitions are too large for the worker: move to a larger, memory-optimized worker type (Glue G.2X+), increase shuffle partitions so each task handles less data, and stop caching what you do not reuse. Driver OOM is different — it almost always means the job pulled a dataset onto the driver with collect() or built a huge broadcast; keep work distributed.

Data skew shows up as stragglers: 199 tasks finish in a minute, one runs for an hour, because one join or group-by key (a null key, a dominant customer) holds a huge share of rows. Fixes: salt the hot key (append a random suffix to spread it across partitions, adjusting the join accordingly), broadcast the smaller side of the join so no shuffle of the big table is needed, or filter/handle the dominant key separately. Adding more executors does not fix skew — one task still holds the hot partition; that distinction is tested.

Small files — thousands of KB-scale objects from streaming delivery — make Spark spend more time on per-file scheduling and S3 requests than on data. Compact them (a periodic Glue compaction job, or grouping options at read time) toward 128 MB–1 GB files.

Also on the checklist: a JDBC source read serially becomes the whole job's bottleneck (partition the read); Lambda transforms failing at exactly 15 minutes have outgrown Lambda; and slow Redshift ELT is often WLM queue waiting or disk-spilling queries, visible in the system views, not raw cluster undersizing.

You expose transformed data to other systems through APIs rather than shared database credentials. The pattern DEA-C01 favors for the warehouse is the Amazon Redshift Data API: an HTTPS API that runs SQL without persistent JDBC connections, authenticates through IAM, and executes asynchronously — you submit a statement, then poll for or fetch results. That connectionless model is exactly what Lambda functions, Step Functions states, and event-driven apps need, because they cannot efficiently hold connection pools. For general-purpose serving, front AWS Lambda with Amazon API Gateway: Lambda queries the store (DynamoDB for key-value lookups at scale, Athena or Redshift for analytics), and API Gateway adds authentication, throttling, and response caching so hot queries do not hammer the backend. Design paginated responses for large result sets rather than streaming unbounded payloads through a synchronous API.

Large language models enter data engineering as a processing step for unstructured and messy data, invoked through Amazon Bedrock. Practical uses the exam can reference: extracting structured fields from free-text documents (turning support tickets or contracts into queryable columns), classifying and tagging records, summarizing long text into analytics-friendly fields, generating embeddings for semantic search, and assisting data-quality work such as entity resolution. Architecturally, treat the LLM as one transformation stage: a Glue job or Lambda iterates records and calls Bedrock, or Bedrock batch inference processes large datasets from S3 asynchronously at lower cost than one-by-one real-time calls. Engineering cautions apply — per-call cost and model rate limits make batching and caching important, retries need backoff, and non-deterministic output means results should be validated against a schema before loading. The signal phrase is unstructured data needing understanding, not just parsing: regex and string functions handle formats; LLMs handle meaning.