Programming Concepts for AWS Data Engineering: IaC, CI/CD, and Lambda Tuning

The fastest way to reduce runtime is to process less data: filter early, read only the columns you need, and prune partitions so the engine never opens irrelevant files. On AWS that starts with layout. A columnar format such as Apache Parquet lets readers skip unneeded columns and use per-row-group min/max statistics to skip blocks. Row formats such as CSV or JSON (gzip-compressed or not) carry no column statistics, so a query that cannot prune partitions reads every file in full. Partition S3 data by the keys queries actually filter on, at a granularity that keeps partitions reasonably large.

Pruning only happens on partition columns. Amazon Athena skips S3 prefixes only when the predicate is on a partition column (for example dt or year/month/day). A filter on a timestamp column stored inside the files does not prune anything, even if it covers the same day, so write queries that filter on the partition columns as well.

AWS Glue offers two read-time partition filters, and they behave differently:

JDBC sources can also push filtering into the source database so less data crosses the network.

Size files at write time before adding compaction. Many tiny objects drown readers in per-file overhead (the small-files problem). Where a producer controls object size, tune it first: Amazon Data Firehose writes one object per buffer flush (per partition, when dynamic partitioning is on), and its S3 buffer interval (up to 900 seconds) and buffer size (up to 128 MiB) decide how large those objects are. A longer buffer gives fewer, larger files at the cost of freshness; a separate compaction job is the fallback when the producer cannot be changed.

Avoid needless shuffles and row-at-a-time code. Wide joins, global sorts and high-cardinality groupBys move data across the network; a broadcast join for a small dimension table avoids shuffling the large side. In Spark, a Python UDF forces every row to be serialized from the JVM to a Python worker and back, and the Catalyst optimizer cannot see inside it. Built-in Spark SQL functions (trim, upper, to_date and so on) run inside the JVM and are optimized, so replacing simple UDFs with built-ins is often the biggest single win. In plain Python, vectorized and batch calls beat per-record loops; in SQL, set-based statements beat cursors. When a job is slow and its data is CSV, unpartitioned or scanned in full, fix the layout and filters before adding workers.

Lambda concurrency is the number of execution environments serving requests at the same moment, and you shape it with three controls that do different jobs: reserved concurrency caps and guarantees, provisioned concurrency pre-warms, and event source mapping settings throttle how fast a poller feeds the function.

Reserved concurrency and the shared pool. An account has a Regional concurrency quota shared by every function without a reservation, and Lambda always keeps at least 100 of it unreserved. Reserving concurrency for one function subtracts it from that unreserved pool around the clock. That guarantees the function capacity, but it also shrinks what everything else can use. Capping a noisy function protects a victim function only if nothing else is also competing for the remainder; giving the critical function its own reservation is the direct guarantee.

Why reserved concurrency alone is the wrong throttle for SQS. The SQS event source mapping scales its pollers up independently of the function. If reserved concurrency is lower than what the pollers want, the extra invocations are throttled. The messages they received go back to the queue after the visibility timeout, their receive count rises, and with a redrive policy those messages can be moved to the dead-letter queue even though no function code ever raised an error. Set maximum concurrency on the event source mapping instead, so the poller never asks for more than the function may run. The limit is per mapping: a function fed by two queues through two mappings can run up to the sum of both limits, so size each mapping for the downstream limit, and any reserved concurrency on the function should be at least that sum.

Invocation type decides which control exists. Amazon S3 event notifications invoke Lambda asynchronously; there is no event source mapping and no maximum-concurrency setting. To cap an S3-triggered function, use reserved concurrency (Lambda queues and retries throttled asynchronous events), or route the notifications through an SQS queue and control the mapping. Pull-based sources (SQS, Kinesis, DynamoDB Streams, Amazon MSK) use event source mappings.

Batching on the mapping. For SQS, batch size and the maximum batching window (0 to 300 seconds) decide how many messages each invocation receives: a batch size of 1 means one invocation per message, so raising both cuts invocation count. The queue's visibility timeout and long polling do not change batch size (Lambda pollers already long-poll). For Kinesis and DynamoDB Streams, batch size and the parallelization factor (concurrent batches per shard) govern throughput.

Provisioned concurrency details. It is configured on a published version or an alias, never on $LATEST, and it only serves callers that invoke that alias or version. An integration that invokes the bare function name runs $LATEST and still gets cold starts while the provisioned environments sit idle, so point every caller at the alias. Application Auto Scaling can schedule provisioned concurrency (more during business hours, less overnight) or track utilization, so you do not pay for warm environments around the clock. Unhandled bursts beyond available concurrency are throttled, which is why buffering with SQS plus a dead-letter queue is the resilient ingestion pattern.

Memory is the one resource setting in Lambda: you choose 128 MB to 10,240 MB (the maximum for every runtime), and CPU and network bandwidth are allocated in proportion. At about 1,769 MB a function gets the equivalent of one full vCPU; higher settings add vCPUs, up to six at 10,240 MB. Single-threaded code can only use one vCPU, so its gains flatten out past about 1,769 MB; only multi-threaded or multi-process code benefits from more. That curve is also a diagnostic: if raising memory toward one vCPU cuts duration sharply, the work is CPU-bound; if duration barely moves, it is waiting on I/O or a downstream service. Raise memory for CPU, not just because the memory graph looks close to the limit. The timeout can be set up to 15 minutes; work that needs longer belongs in AWS Glue or Amazon EMR.

Reuse the execution environment. Code outside the handler (SDK clients, database connections, reference data loaded into memory) runs once when the environment initializes and is reused by every warm invocation in that environment. Files written to /tmp also survive between warm invocations of the same environment, so a cache there is valid as long as you treat it as optional.

Choosing storage inside a function (Skill 1.4.7):

Mounting EFS has prerequisites. The function must be attached to the VPC (subnets in the Availability Zones that have mount targets), it connects through an EFS access point, and the local mount path must start with /mnt/. Security groups must allow NFS on TCP port 2049 from the function to the mount targets, and the execution role needs the EFS client permissions. Mount targets have IP addresses inside the VPC, so a VPC-attached function reaches EFS directly: no NAT gateway and no S3 endpoint are involved. A NAT gateway is only needed when the function must reach the internet or public AWS endpoints.

Pick the language the runtime favors, not a personal favorite. Python is the default: AWS Glue PySpark and Python shell jobs, Lambda functions, Airflow DAGs on Amazon MWAA, and AWS SDK (boto3) automation are all Python-first, and pandas and AWS SDK for pandas cover single-node transformation. SQL is the second universal language: Amazon Athena and Amazon Redshift are SQL engines, Spark SQL expresses most Glue transformations, and when data is already loaded into Redshift, transforming it with SQL in place (ELT) is usually simpler and faster than exporting it to another engine.

Scala and Java matter because Apache Spark runs on the JVM. Be precise about Glue: an AWS Glue Spark job script is written in Python (PySpark) or Scala only. Java and R have Spark APIs, and you can run Java or R Spark applications on Amazon EMR, but neither is a Glue job script language. Scala avoids the Python-UDF serialization cost, so performance-critical Spark logic is sometimes written in Scala. Java also appears in Kafka clients around Amazon MSK and in the Kinesis Client Library. R is a statistics and analysis language (available on EMR through SparkR and R kernels in EMR Studio notebooks), not a pipeline-orchestration language.

Bash (and PowerShell on Windows hosts) scripts the AWS CLI and host setup. The classic example is an EMR bootstrap action: a script, usually Bash, that runs on every node as it launches (including nodes added later by scaling) before applications start, which makes it the right place to install OS packages and Python libraries. A Spark step, by contrast, runs one application on the cluster after it is up and does not prepare new nodes. Bash also drives container entrypoints and CI/CD build steps.

The selection logic: a Glue or Lambda transformation suggests Python; maximum Spark performance for UDF-heavy logic suggests Scala; ad hoc analysis over S3 data suggests SQL in Athena; transformation of data already in Redshift suggests SQL there; node setup or CLI automation suggests Bash. Frameworks follow the same logic: Spark for distributed batch, pandas for single-node work that fits in memory, SQL engines when the logic is expressible as queries.

Treat pipeline code like production software: versioned, reviewed, tested, logged and monitored. Version control holds everything (Glue scripts, Lambda handlers, Airflow DAGs, SQL, and the IaC templates that deploy them) in Git, with pull-request review before merge to the main branch. The repository, not whatever happens to be deployed, is the source of truth. That shapes rollback: to undo a bad change, git revert the commit through a reviewed pull request and let the pipeline redeploy. Restoring an older S3 object version or editing the deployed script by hand puts the environment out of step with Git, and the next merge redeploys the bug.

Testing has layers, each catching a different class of fault:

Mocks in a unit test cannot reveal a missing IAM permission or a wrong catalog schema; only a test against deployed resources can. Glue scripts that import awsglue (GlueContext, DynamicFrame) can still be unit-tested in CI without paying for Glue capacity by running them in the official AWS Glue container image (aws-glue-libs), which ships the Glue libraries and Spark locally.

Logging centers on Amazon CloudWatch Logs. Write structured (JSON) log lines so CloudWatch Logs Insights can query them, and include a correlation ID (for example the record or file ID) on every line, so one record can be followed across many concurrent invocations and stages. Log record counts and durations at each stage boundary as well, so a run that silently drops rows shows up as a mismatch between stages. CloudWatch Lambda Insights is different: it collects performance metrics such as CPU, memory and cold starts per function, not per-record traces.

Monitoring turns signals into action. For custom metrics from Lambda, the CloudWatch embedded metric format (EMF) writes metrics as specially structured log lines that CloudWatch extracts asynchronously, so a high-throughput function emits per-source metrics without making PutMetricData API calls. For job outcomes, AWS Glue publishes Glue Job State Change events to Amazon EventBridge; a rule matching the FAILED (or TIMEOUT) state can route them to an Amazon SNS topic that emails the on-call engineer. Alarms on Lambda errors and throttles, Step Functions failed executions and SQS queue age complete the picture. AWS X-Ray traces requests across Lambda and API-fronted services when you need to see where latency lives.

All three tools deploy through the same engine, AWS CloudFormation, so the choice is about authoring experience, compared below.

Why IaC: repeatable resource deployment. One parameterized template creates identical dev, test and prod stacks; changes are code-reviewed and previewed before execution; a failed update rolls back. When an environment must be recreated identically in another account or Region, the answer is a template, never manual console steps.

CloudFormation controls that are easy to confuse:

With AWS Organizations, a StackSet using service-managed permissions targets an organizational unit, and automatic deployment creates the stack in every current account and in any account added to the OU later (and can remove it from accounts that leave). StackSets distribute infrastructure; they do not run tests or approval gates, so they are not a CI/CD pipeline. For repetition inside one template, plain YAML has no loops, but the AWS::LanguageExtensions transform adds Fn::ForEach, which expands one resource definition over a list (for example one bucket per data domain).

CDK in practice. The CLI lifecycle is: cdk bootstrap (once per account and Region, creates the assets bucket and deployment roles), cdk synth (produce the CloudFormation template), cdk diff (compare the synthesized template with the deployed stack, showing which resources and properties will change), cdk deploy, and cdk watch (fast redeploys on file save, for development). Constructs come in three levels: L1 (Cfn* classes, one-to-one with CloudFormation resources), L2 (curated resources with defaults and helper methods, such as Bucket) and L3 (patterns that combine several resources). When an L2 construct does not expose a property yet, use an escape hatch: reach the underlying L1 resource through node.defaultChild and set the value with a property override, keeping the rest of the L2 behaviour. Loops and conditionals come from the programming language itself.

AWS SAM is the guide's named tool for packaging and deploying serverless data pipelines: a SAM template adds the AWS::Serverless-2016-10-31 transform and short resource types (AWS::Serverless::Function, ::StateMachine, ::SimpleTable, ::Api), and the SAM CLI builds, tests, packages and deploys them as a CloudFormation stack.

Permissions without hand-written policies. SAM policy templates attach scoped IAM policies by name and a resource parameter, for example DynamoDBReadPolicy with a table name or StepFunctionsExecutionPolicy with a state machine name. Connectors (AWS::Serverless::Connector, or the Connectors property) go further: you declare that a source resource needs Read or Write access to a destination in the same template, and SAM generates the least-privilege policy.

Gradual deployments. Setting AutoPublishAlias on a function makes SAM publish a new version and move an alias on every code change; adding DeploymentPreference hands the alias shift to AWS CodeDeploy with a canary or linear type (for example 10 percent first, the rest after a wait) and a list of CloudWatch Alarms that roll traffic back automatically if they fire. Pre- and post-traffic hook functions can run validation tests during the shift.

Why packaging uses content-hashed keys. CloudFormation updates a Lambda function's code only when the template's code location changes. If a build uploads a new zip to the same S3 bucket and key every time, the template is unchanged and the stack update leaves the old code running. sam package/sam deploy and aws cloudformation package upload artifacts under keys derived from the content hash, so each new build changes the template and the update takes effect.

CI/CD automates the path from commit to production: continuous integration builds and tests every change, and continuous delivery deploys it through environments with gates. On AWS the reference toolchain is a Git repository feeding AWS CodePipeline, with AWS CodeBuild running build and test stages and deployment through CloudFormation, cdk deploy or sam deploy; the concepts transfer to GitHub Actions and similar tools.

A data-pipeline flow has distinct stages. CI: on each commit, lint, run unit tests on transformation logic, synthesize or validate the IaC, and package artifacts once (a Lambda bundle, a Glue script uploaded to S3, a container image in Amazon ECR). Test environment: deploy the stack, run the pipeline on representative sample data, and assert on output (row counts, schema, data-quality rules). Production: deploy the same artifacts after the checks pass.

Build once, promote the same artifact. The artifact that passed testing is the one that reaches production; only parameters differ by stage (bucket names, connection secrets referenced from AWS Secrets Manager). Rebuilding per environment, or keeping per-environment branches with hard-coded values, means production runs something that was never tested.

Gates before production. A CodePipeline production stage can chain three actions: a create change set action (CHANGE_SET_REPLACE) computes exactly what will change, a manual approval action pauses the pipeline so a reviewer can inspect it, and an execute change set action (CHANGE_SET_EXECUTE) applies it. Place the approval immediately before the production deploy stage. A single create-or-update deploy action applies changes immediately, leaving no point at which a person sees the change set.

Safe rollout and rollback. CloudFormation rolls a failed update back automatically; for Lambda, SAM gradual deployments shift traffic between versions with alarm-based rollback; and a bad change in code is undone by reverting the commit and letting the pipeline redeploy. Deploying code and infrastructure from the same repository keeps them in step: a Glue job and the IAM role it needs change in one reviewed commit. Copying scripts to S3 by hand or clicking through the console for each release is what a CI/CD pipeline with automated tests and IaC-driven deployment replaces: every release becomes repeatable, tested and traceable to a commit.

Distributed computing means splitting one job across many machines that work in parallel and combine results, the model behind Apache Spark on AWS Glue and Amazon EMR. Data is divided into partitions, and each stage runs one task per partition on executor cores. Operations that stay within a partition (map, filter) scale out cheaply; operations that regroup data across machines (join, groupBy, sort) trigger a shuffle, network movement that is usually the bottleneck.

Parallelism comes from partitions, not from workers alone. If a stage has 16 partitions it runs 16 tasks, so adding workers beyond 16 cores leaves them idle. Use repartition(n) to raise the partition count (it performs a full shuffle) and coalesce(n) to lower it cheaply, for example before writing fewer output files. Input parallelism comes from splits: a large Parquet or uncompressed file can be split into many tasks, but a single gzip file is not splittable, so one task reads the whole file however many workers the job has. Split the input into many files at the source, or use a splittable format.

Other consequences to recognise: data skew, where one oversized key makes a single task the straggler; start-up and coordination overhead, which lets a single-node tool (pandas, a Lambda function) beat a cluster on small data; and resilience through lineage, where lost partitions are recomputed instead of restarting the job. Long-running streaming jobs add checkpointing, saving progress (offsets and state) to durable storage such as Amazon S3 so a restarted job resumes where it stopped instead of reprocessing or skipping data. Scaling out handles more data; scaling up helps memory-hungry tasks; neither fixes skew or an unnecessary shuffle.

Graphs and trees. A graph is nodes connected by edges. Spark plans and orchestration workflows (Airflow DAGs, Step Functions) are directed acyclic graphs: edges point one way and no path loops back, which guarantees a topological order in which every task runs after its dependencies. A cycle would make a task wait on itself, so there would be no valid order. A tree is a special case with one root where every other node has exactly one parent; a DAG allows a node with several parents (a join of two upstream tasks). Graphs also model lineage and relationship-heavy data.

Hashes versus sorted trees. A hash structure (hash map or hash set) gives near constant-time lookup by exact key, which suits deduplication ("have I seen this event ID?") and key-value access as in Amazon DynamoDB partition keys and hash partitioning. It has no order, so it cannot answer range queries. A sorted tree such as a B-tree keeps keys in order, so it answers both exact lookups and ranges ("every reading between two timestamps, in order") in logarithmic time, which is why relational indexes use it. Expect to match a structure to an access pattern, not to implement one.

Scenario: a team hand-built a pipeline in the console. An SQS queue receives S3 event notifications and triggers a Lambda function that validates each file; a Step Functions state machine runs a Glue job; results land in a DynamoDB table that tolerates limited write concurrency. Management wants identical dev and prod environments, reviewed changes, tested releases and faster processing. What changes?

Step 1: codify. Define the whole stack (function, queue and event source mapping, state machine, Glue job, table, IAM) in an AWS SAM template, using policy templates or connectors for permissions, and keep it with all code in Git. Console-built resources are replaced by deployed stacks, so environments cannot drift apart by construction.

Step 2: automate. CodePipeline runs on merge: CodeBuild runs unit tests (Glue scripts in the Glue container image), sam build packages once, the stack deploys to a test account, and an integration test drops a sample file and checks the table. A change set, a manual approval and an execute-change-set action promote the same artifact to production, where AutoPublishAlias and a canary deployment preference with alarms shift traffic gradually.

Step 3: tune. Profiling shows validation is CPU-bound, so memory goes up toward one vCPU (about 1,769 MB) and duration drops while cost barely moves. To protect the table, the event source mapping gets a maximum concurrency limit rather than relying on reserved concurrency alone, so messages wait in the queue instead of being throttled toward the dead-letter queue; batch size and batching window reduce invocation count. Logs carry the file ID as a correlation ID, and a Glue job state-change rule in EventBridge alerts an SNS topic on failure.

Each choice maps to a Task 1.4 skill: IaC for repeatable deployment, SAM for serverless packaging, CI/CD with testing, software-engineering practice, and Lambda concurrency and performance configuration.