DVA-C03 Lambda Development: Configuration, VPC Access, Error Handling and Event Sources

Lambda runs your code by calling one entry point, the handler, inside an execution environment that it creates, reuses for later invocations and eventually discards. The runtime setting chooses the language environment (a managed runtime such as Python, Node.js, Java or .NET, or an OS-only runtime such as provided.al2023 for Go, Rust or a custom runtime), and the handler setting tells that runtime which function to call.

The lifecycle: init, invoke, reuse

A new environment first runs the init phase: it starts the runtime, starts any extensions and runs all code outside the handler (imports, module-level variables, SDK clients). That one-off cost is the cold start. Each invocation then runs only the handler. After an invocation, Lambda keeps the environment warm and may send it the next request, so:

• Objects created outside the handler (SDK clients, database connections, loaded configuration) are reused by later invocations in the same environment. Moving expensive setup out of the handler is the standard way to cut warm-invocation duration.
• Module-level variables keep their values between invocations in that environment, so a global list that you append to keeps growing. Per-request state belongs inside the handler.
• Files written to /tmp stay there for the life of the environment. /tmp is not shared between concurrent environments, and Lambda never saves memory state to it for you.

Lambda never promises reuse: each concurrent request gets its own environment, and environments are recycled at any time. Treat reuse as an optimisation (a cache), never as storage. For Java, Python and .NET, Lambda SnapStart is another cold-start option that resumes new environments from a snapshot of an initialised one.

Memory is the one performance dial in Lambda: you set memory from 128 MB to 10,240 MB, and Lambda allocates CPU in proportion to memory. At 1,769 MB a function gets the equivalent of one full vCPU, and more memory gives more vCPUs. A CPU-bound function that uses little of its memory still speeds up when you raise the memory setting, often for a similar or lower cost because it finishes faster.

Match the symptom to the setting. A No space left on device error while writing a file means /tmp is too small: raise ephemeral storage; raising memory does not enlarge /tmp, and a layer cannot add disk. A Task timed out after 900.00 seconds error means the job hits the hard 15-minute ceiling, which no setting raises. Split the work into smaller invocations (for example, an AWS Step Functions workflow that runs batches) or move it to a container on Amazon ECS with AWS Fargate. More memory only helps if the time is spent on CPU, not waiting on an API.

Environment variables are key-value settings stored in the function's configuration and read by your code at runtime (for example os.environ['TABLE_NAME']). They let one deployment package run in development, test and production with a different bucket name, table name or log level in each, with no rebuild. Lambda also sets reserved variables of its own, such as AWS_REGION and AWS_LAMBDA_FUNCTION_NAME.

• Size: all of a function's environment variables together are limited to 4 KB. The limit cannot be raised, and splitting a large document across several variables does not help because the total is what counts.
• Encryption: Lambda encrypts environment variables at rest with an AWS managed KMS key by default; you can choose a customer managed key instead to control its key policy and rotation. Encryption helpers in the console can also encrypt values client-side so they are hidden in the console, and the code then decrypts them. Secrets are still better kept in Secrets Manager than in variables (see the next sections).

Larger or frequently changed configuration

When configuration is too large for 4 KB, or must change without touching the function at all, keep it outside the function and read it at runtime: an AWS Systems Manager Parameter Store parameter (standard parameters hold 4 KB, advanced parameters 8 KB), AWS AppConfig read through the AppConfig Agent Lambda extension, or AWS Secrets Manager for secrets. A file inside a layer or the package is not changeable without a new deployment.

Versions freeze configuration

Publishing a version takes an immutable snapshot of the code and the configuration, including environment variables, of $LATEST. You cannot edit a published version. An alias (such as prod) is a named pointer to a version; clients invoke the alias ARN, so you promote or roll back by moving the alias. Editing a variable in the console changes only $LATEST. To get it into production safely, publish a new version and point the alias at it; pointing the alias at the old version again is the rollback. Keep production aliases on published versions so every release has a fixed, known rollback point.

Concurrency is the number of requests a function is handling at the same moment, and therefore the number of execution environments it is using. All functions in a Region share the account's concurrency quota. Two controls look alike but solve different problems:

Use reserved concurrency to protect a downstream system that can take only so many simultaneous connections, while every other function keeps scaling, or set it to 0 to stop a runaway function at once. Use provisioned concurrency on an alias for latency-sensitive APIs, and schedule it with Application Auto Scaling for predictable peaks such as the start of the working day.

One trap: when the trigger is an SQS event source mapping, a low reserved concurrency makes Lambda throttle the batches the poller has already received. Those messages return to the queue, each receive counts toward the redrive policy, and valid messages can land in the dead-letter queue unprocessed. To cap an SQS-triggered function, set maximum concurrency on the event source mapping instead.

A Lambda layer is a .zip archive of libraries, a custom runtime or other files that you publish once and attach to many functions, so shared code is maintained in one place and each function's package stays small. A function can use up to five layers. Each layer version is immutable: publishing a change creates a new version number, and functions keep the version they reference until you update them. Layer versions can be shared with other accounts through layer permissions.

Lambda extracts layer contents into /opt, and each runtime only searches specific folders there. The folder structure inside the .zip therefore decides whether imports work:

A Python library placed at the root of the .zip lands directly in /opt, which is not on the Python path, so imports fail with ModuleNotFoundError. Rebuild the layer with the package under python/.

Container images take no layers. A function deployed as a container image cannot attach layers at all. When migrating, copy what the layers held (shared code and any extension) into the image during the Docker build.

Extensions

A Lambda extension is a companion process that plugs into the execution environment's lifecycle to add monitoring, security or configuration tooling without changing handler code. External extensions run as separate processes (delivered as layers, under /opt/extensions), start during init, can keep working after the function returns its response (for example, flushing telemetry), and are typical for observability vendors' agents. Internal extensions run inside the runtime process (through language-specific environment variables or wrapper scripts).

The AWS Parameters and Secrets Lambda Extension is the AWS-supplied extension for caching Parameter Store parameters and Secrets Manager secrets inside the environment. The code calls a local HTTP endpoint (port 2773 by default) and passes the function's session token in the X-Aws-Parameters-Secrets-Token header; the extension fetches the value on a miss and serves it from cache until its time-to-live expires, which cuts API cost, latency and throttling. Key points:

• It calls Secrets Manager and Parameter Store with the function's execution role, so the role needs secretsmanager:GetSecretValue (or ssm:GetParameter, plus kms:Decrypt for a customer managed key). An access-denied answer from the local endpoint means the role lacks that permission; a resource policy for lambda.amazonaws.com does not help.
• Cache freshness is set with environment variables: SECRETS_MANAGER_TTL and SSM_PARAMETER_STORE_TTL (seconds; default 300). Use a short TTL for a secret that rotates often.

To reach resources that exist only inside your VPC (an RDS database, an ElastiCache cluster, an internal API), you attach the function to that VPC by giving it subnets and security groups. By default a function runs in a Lambda-managed network with internet access but no route into your VPC, so connections to private endpoints time out. Adding IAM permissions or a resource-based policy never fixes a network path.

• Execution role permissions. Lambda creates elastic network interfaces (Hyperplane ENIs) in your subnets, using the function's execution role. The role needs ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces and ec2:DeleteNetworkInterface, which the AWS managed policy AWSLambdaVPCAccessExecutionRole grants among others (it also covers subnet description, private IP assignment and CloudWatch Logs). Without them, saving the VPC configuration fails. AWSLambdaBasicExecutionRole covers only logging.
• Security groups. Allow the function into the target by adding an inbound rule on the target's security group (on the database or service port) whose source is the function's security group. A security-group reference follows the function's network interfaces wherever they are created, so the rule never needs updating as addresses change. Security groups are stateful, so no extra outbound rule is needed on the database for replies.
• Subnets. Choose private subnets in at least two Availability Zones for resilience.

Outbound access from a VPC-attached function

Once attached to a VPC, the function's traffic follows that VPC's routing. Lambda's ENIs never receive public IP addresses, so placing the function in a public subnet with an internet gateway route does not give it internet access, and enabling auto-assign public IPv4 on the subnet does not change that.

An interface endpoint is for AWS (or PrivateLink partner) services; it cannot reach an arbitrary public API. When a security team forbids any internet egress, endpoints are the answer and NAT is ruled out.

Relational databases: RDS Proxy

Each concurrent execution environment opens its own database connections, so a function that scales to hundreds of environments can exhaust a database's connection limit. Amazon RDS Proxy sits between the function and the RDS or Aurora database, pools and shares connections, and absorbs bursts without changing the database engine or instance size; the code change is only the endpoint it connects to. Reusing a connection opened outside the handler helps per environment but does not cap the total across hundreds of environments; reserved concurrency caps it, but at the cost of throughput.

By default, RDS Proxy reads the database credentials from AWS Secrets Manager for its own connections to the database; since 2025 it can instead use end-to-end IAM authentication, connecting to the database with IAM and needing no secret. Either way, the function can authenticate to the proxy with IAM authentication: enable IAM auth on the proxy, grant the execution role rds-db:connect, and connect with a short-lived token generated by the SDK (for example generate_db_auth_token), so the function never handles the password. (The RDS Data API, an HTTPS alternative to connections, works with Aurora only.)

Every Lambda invocation is one of three kinds, and the kind decides who waits, who retries and which error-handling features apply.

The third InvocationType, DryRun, only checks that the caller is allowed to invoke and runs nothing. InvokeWithResponseStream streams a response back progressively and is still synchronous.

Calling a function from code

Choose Event when the caller should continue as soon as Lambda accepts the request (fire-and-forget): the caller stops paying for idle waiting, and Lambda takes over the retries. Choose RequestResponse when the caller needs the result.

A long synchronous call needs the SDK client configured for it. AWS SDKs have their own read timeouts (boto3's default is 60 seconds) and automatic retries. If the function runs longer than the client's read timeout, the client gives up and retries the call, starting a second run while the first is still going. Set the client's read timeout above the function's longest run and turn off retries for that call:

from botocore.config import Config
import boto3

cfg = Config(read_timeout=310, retries={"total_max_attempts": 1})
client = boto3.client("lambda", config=cfg)
client.invoke(FunctionName="invoice-batch", Payload=b"{}")

(Here the function's own timeout is 300 seconds, so the client waits slightly longer.)

Raising the function's own timeout does not help here: the function was not timing out; the client was.

For an asynchronous invocation, Lambda puts the event in an internal queue, invokes the function, and on an error retries it up to two more times by default; an event that waits too long is discarded. You tune both on the function's asynchronous invocation configuration: maximum retry attempts (0–2) and maximum event age (60 seconds to 6 hours, default 6 hours). To learn about a failure quickly, lower both: fewer retries and a shorter maximum event age move an undeliverable event to the failure target sooner, so the worst-case delay is bounded by the event age you set. Because events can be delivered more than once, asynchronous handlers should be idempotent.

What happens to an event that finally fails depends on two features that overlap:

AWS recommends destinations for new work because they carry more context and more targets. Three rules decide most scenarios:

• Function destinations and DLQs apply only to asynchronous invocations. A synchronous RequestResponse call never triggers them, so a caller that waits must switch to InvocationType=Event before an on-failure destination will fire. They are configured on the function that fails, not on its caller.
• Lambda writes to the destination or DLQ with the function's execution role. The role needs sqs:SendMessage, sns:Publish, events:PutEvents or lambda:InvokeFunction on the target. When a DLQ delivery fails, the function's DeadLetterErrors metric rises; the usual cause is a missing permission on the execution role (or KMS access for an encrypted queue).
• Polled sources do not use these settings. SQS, Kinesis and DynamoDB Streams handle failures on the event source mapping (next section), and the queue's own visibility timeout or redrive policy has nothing to do with an asynchronous invocation.

An event source mapping is a Lambda resource that polls a queue or stream, gathers records into batches and invokes the function synchronously with each batch. It is how SQS, Kinesis and DynamoDB Streams trigger a function; no EventBridge rule or resource-based policy is involved. The execution role needs permission to read the source; AWS managed policies such as AWSLambdaSQSQueueExecutionRole and AWSLambdaDynamoDBExecutionRole (GetRecords, GetShardIterator, DescribeStream, ListStreams) supply it. Without it, creating the mapping fails.

Amazon SQS

• Deletion is automatic. When the function returns success, the mapping deletes the batch's messages. If the function throws, the whole batch becomes visible again after the visibility timeout and is reprocessed.
• Partial batch responses. Add ReportBatchItemFailures to the mapping's function response types and return {"batchItemFailures": [{"itemIdentifier": "<messageId>"}]}. Only the listed messages return to the queue; the rest are deleted. Batch size stays as large as you want.
• Visibility timeout. Set the queue's visibility timeout to at least six times the function timeout (plus any batching window). A shorter value lets a message reappear and be processed a second time while the first run is still working.
• Failed messages. Isolate poison messages with a redrive policy on the source queue that moves a message to a dead-letter queue after maxReceiveCount receives. The function's asynchronous DLQ and destinations do not apply to SQS mappings.
• Throttling. Cap concurrency with the mapping's maximum concurrency (see the concurrency section), not with reserved concurrency.

Kinesis Data Streams and DynamoDB Streams

Stream mappings read each shard in order. If a batch fails, Lambda retries that same batch, and by default it keeps retrying until the records expire from the stream, so the shard is blocked while newer records pile up (the IteratorAge metric climbs). The mapping offers these controls:

These settings live on the event source mapping. Configuring retries or an on-failure destination in the function's asynchronous invocation settings has no effect on a stream, which is why a shard can stay blocked even though "a destination is configured". Note what the failure record contains: an SQS or SNS on-failure destination receives metadata only (the stream ARN, shard ID and start and end sequence numbers), not the changed data. To reprocess, read the records from the stream with those positions before they expire (DynamoDB Streams keep records for 24 hours). An S3 on-failure destination also stores the batch contents.

Wiring DynamoDB to Lambda

To run code when items change, enable DynamoDB Streams on the table, then create an event source mapping from the stream to the function. The stream's view type decides what each record carries:

The view type cannot be changed on an existing stream. To change it, disable the stream and enable a new one with the new view type; the new stream has a new stream ARN, so the mapping must be recreated against it. Choose the view type up front from what the consumer needs, before or after images, or both.

Choose the starting position when creating a stream mapping: TRIM_HORIZON (oldest available record) or LATEST (new records only); Kinesis also supports AT_TIMESTAMP. Add filter criteria to invoke the function only for matching records, for example inserts only:

{ "Filters": [ { "Pattern": "{\"eventName\": [\"INSERT\"]}" } ] }

Filtering on the mapping means the function is never invoked for non-matching records, unlike checking eventName inside the handler, which still pays for every invocation.

With Lambda proxy integration, API Gateway passes the whole HTTP request (path, query string, headers, body) to the function as the event and turns the function's return value directly into the HTTP response. The function is therefore responsible for the response format. For a REST API it must return an object like this:

return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json",
                "Access-Control-Allow-Origin": "https://shop.example.net"},
    "body": json.dumps({"message": "ok"}),
}

• body must be a string (serialise JSON yourself); isBase64Encoded is set for binary bodies. Returning a bare dictionary, a plain string or other keys produces a 502 Bad Gateway with a "malformed Lambda proxy response" error.
• CORS: enabling CORS in the console configures the OPTIONS preflight (a mock integration). With proxy integration there is no integration response to map headers on, so the Access-Control-Allow-Origin header on real GET/POST responses must come from the function's returned headers.
• REST API integrations time out at 29 seconds by default, so long work belongs behind an asynchronous pattern.

Permission to invoke

API Gateway invokes the function under the function's resource-based policy, which needs a statement allowing lambda:InvokeFunction for the principal apigateway.amazonaws.com, normally scoped with a source ARN for the API. The console adds this automatically when you pick a function; the AWS CLI, SDKs and some templates do not, and calls then fail with a 500 and "Invalid permissions on Lambda function" in the execution logs. The fix is aws lambda add-permission (or the template's permission resource). The execution role is the opposite direction: it is what the function itself may call, and it never authorises callers.