SAP-C03 Event-Driven, Serverless and Multi-Tenant Architecture

Most AWS messaging services deliver at least once, so an event-driven architecture is correct only when every consumer can safely receive the same message twice. Amazon SQS standard queues, Amazon SNS standard topics and Amazon EventBridge all promise at-least-once delivery and do not promise order. Ordering and duplicate suppression are opt-in features with a throughput cost, so a Professional-level design states which entity needs ordering and applies it at exactly that granularity.

Idempotent consumers

You cannot make delivery exactly-once by turning retries off: EventBridge can still deliver an event twice with a zero retry policy, and serialising a consumer with reserved concurrency of one only changes when the duplicate runs. The durable fix lives in the consumer. Pick a stable idempotency key (the event ID, order ID or transaction ID), and before the side effect claim it with a conditional write, for example a DynamoDB PutItem with attribute_not_exists(pk). The first delivery wins the condition and performs the charge; a duplicate fails the condition and returns quietly. Retries stay switched on, so no event is lost. Archive and replay is a recovery tool, not a deduplication tool: replayed events are new deliveries the consumer must also tolerate.

Ordering and deduplication with SQS FIFO

An SQS FIFO queue orders messages within a message group, not across the whole queue. Use one group ID per entity that needs ordering (a merchant, account or order) and consumers can work on many groups in parallel while each group stays in order. One fixed group ID for the whole queue serialises everything, so only one message is in flight and throughput cannot grow. FIFO queues also drop a repeated send that carries the same deduplication ID within the five-minute deduplication interval; set it to the business key (the transaction ID) or turn on content-based deduplication, which hashes the body.

Ordering with Kinesis partition keys

Amazon Kinesis Data Streams orders records per shard, and the partition key decides the shard. Records with the same key always land on the same shard in order; a random key spreads load but scatters an entity's records, so their order is lost. Kinesis does not deduplicate a retried PutRecord, so consumers still need idempotency.

The key must be the narrowest unit that needs ordering. A key coarser than the ordering requirement (a warehouse code when only each parcel's scan events must stay in order, say) becomes a hot partition key: every record for that value hashes to one shard and hits that shard's write limit even while the stream as a whole is mostly idle. The rule is that capacity follows keys, not shard counts: any change that keeps one busy value on one hash maps it to one shard, whatever the stream's size or capacity mode, and any change that randomises the hash (a random key or a random ExplicitHashKey, which overrides the key) restores throughput by giving up order. Only a finer key that still satisfies the ordering requirement (here, the parcel ID) gives both.

An SNS FIFO topic orders and deduplicates per message group and can deliver to both SQS FIFO and SQS standard queues, but strict order and deduplication survive only to FIFO queues; a standard queue on the end quietly reverts to best-effort order and at-least-once delivery. A single shared queue polled by three services does not fan out at all: each message is received by one consumer and deleted.

SQS with Lambda: visibility timeout and partial batch responses

Many duplicate-processing problems with SQS are self-inflicted by configuration, not by the at-least-once contract. Two settings matter when SQS is a Lambda event source.

• Visibility timeout. While a consumer works on a message it is hidden; if processing outlasts the visibility timeout, the message reappears and another invocation processes it again. AWS recommends a visibility timeout of at least six times the function timeout for a Lambda event source. Shortening the function timeout to match a short visibility timeout simply makes long jobs fail.
• Partial batch responses. By default, if one message in a batch fails, the whole batch returns to the queue and the successful messages are processed again. Turning on ReportBatchItemFailures lets the function return only the failed message IDs, so Lambda deletes the rest.

A redrive policy (maxReceiveCount plus a dead-letter queue) isolates messages that keep failing, but it does not stop redelivery and a higher receive count allows more of it. Even with both settings right, a standard queue can occasionally deliver twice, so the consumer should still be idempotent.

Choose the service by the routing, replay and consumption model the scenario needs; each one is the right answer for a different combination.

Typical signals: content-based routing to many consumers, SaaS targets and replay for a new consumer point to an EventBridge custom bus with one rule per consumer, API destinations and an archive. Several independent readers, re-reading recent history and per-key order point to a Kinesis data stream (24-hour retention by default, extendable). Producers should never change when a consumer is added, so copying a message to a queue per consumer from the producer is a coupling smell.

Delivering to SaaS partners with API destinations

An API destination is an EventBridge target that calls an HTTPS endpoint through a connection that stores the authorization (Basic, API key or OAuth client credentials, where EventBridge obtains and refreshes the token). It has an invocation rate limit per second, which throttles delivery to what the partner accepts. To survive a partner outage of several hours without losing events, set the rule target's retry policy (maximum event age up to 24 hours and up to 185 retry attempts) and attach an SQS dead-letter queue for anything still undelivered. SNS HTTPS subscriptions can throttle and retry but cannot fetch OAuth tokens, and a Lambda forwarder means writing and operating custom token and retry code.

A stateless application keeps no user or job state on the instance, container or function that serves the request, so any copy can serve any request and copies can be added, replaced or removed at any time. State moves to a managed store chosen by its shape:

The test for any proposed fix is simple: does the state still live on one instance? Anything that only keeps a user pinned to an instance, or keeps an instance alive a little longer before it goes, postpones the loss; a failed health check or a scale-in event still takes that instance's state with it. Only moving the state out removes the risk, and once it lives in a shared store, instance affinity is no longer needed.

Containers follow the same rule, because a task's own storage has the task's lifetime: Fargate ephemeral storage disappears with the task however large it is, and an Amazon EBS volume that ECS attaches to a task is deleted when that task stops, by default. Deployment safety features protect against bad releases, not against ordinary scale-in. So large client uploads go straight to S3 with presigned URLs (which also keeps big transfers off the service), and progress goes into a DynamoDB item per job. (Container scaling and orchestration themselves are covered in the SAP-C03 containers, auto scaling, ingress and edge lesson.)

A serverless data pipeline ingests records with Kinesis Data Streams, processes them in Lambda, and lands them in S3 for Glue and Athena to transform and query, with no servers to manage at any stage. Lambda reads a Kinesis stream through an event source mapping, which by default processes one batch per shard at a time, in order, and retries a failing batch until the records expire. That default creates the two classic problems: lag and stuck shards.

• Lag while invocations are fast means too little concurrency per shard. The parallelization factor (1 to 10) runs up to ten batches per shard concurrently, and Lambda still keeps records with the same partition key in order. A bigger batch size or batching window adds work per invocation without adding parallelism.
• A shard stalled for hours is a poison record retried until it ages out. Turn on bisect batch on function error to split the failing batch and isolate the bad record, cap maximum retry attempts and maximum record age so the shard moves on, and add an on-failure destination (SQS, SNS or S3) that keeps the details of what was skipped.

Shared reads versus enhanced fan-out

Standard consumers share each shard's read throughput (2 MB per second per shard, across all of them). Adding a fifth or sixth polling consumer raises IteratorAge for everyone and produces ReadProvisionedThroughputExceeded even when writes are low. More shards add capacity per shard but every consumer still polls every shard. Enhanced fan-out registers a consumer that gets its own 2 MB per second per shard, pushed over HTTP/2 with SubscribeToShard; point the Lambda event source mapping at the consumer ARN to give a latency-critical function dedicated throughput without slowing the others.

Capacity mode is a separate choice: on-demand mode scales shards automatically for unpredictable traffic, provisioned mode is cheaper for steady, known throughput. Neither fixes a hot partition key.

Serverless analytics on S3 with AWS Glue and Amazon Athena

A serverless analytics pipeline lands raw data in Amazon S3, describes it in the AWS Glue Data Catalog, transforms it with Glue ETL jobs (Spark) and queries it with Amazon Athena, which bills on-demand queries by bytes scanned. Cost and speed therefore come from scanning less.

• Columnar formats. Converting JSON or CSV to Apache Parquet (or ORC) with a Glue job means a query that reads 5 of 120 fields reads only those columns, compressed.
• Partitioning. Partitioning the table by the column most queries filter on (usually event date) lets Athena skip whole prefixes; queries must filter on the partition column to benefit.

When the goal is a cheaper or faster query, ask of every option whether it reduces the bytes read. Changes to how compute is purchased, how often the catalog is refreshed, or where objects are stored can matter for other reasons, but only format, compression and partition pruning shrink the scan.

Highly partitioned tables: projection versus registration

Projection fits predictable layouts with very many partitions (date, hour and a known set of device types). The governance and lakehouse side of data design (Lake Formation, table formats) belongs to the data, storage and database architecture lesson.

AWS Step Functions runs multi-step workflows as state machines whose progress the service persists, so a workflow survives failures and can wait for days. The first decision is the workflow type.

Decide from three facts in the scenario. Duration: anything that can outlast five minutes (a loan application waiting on a credit bureau, a claim waiting on an adjuster) must be Standard. Side effects: a step that must not run twice (issuing a refund, provisioning a licence) needs Standard's exactly-once steps, or an idempotent design if Express is chosen. Volume and caller: very high rates of short, idempotent work (enriching IoT telemetry, transforming clickstream records) suit Express; use asynchronous Express when the caller does not need the result and synchronous Express when it waits for the output, as an API backend does. A home-made chain of functions and queues gives at-least-once behaviour and no built-in execution history.

Callbacks, heartbeats and sagas

• Task token callback (.waitForTaskToken): the task passes a token to an external system (a queue, a web app, an on-premises worker) and the execution pauses, without polling and without paying for transitions, until that system calls SendTaskSuccess or SendTaskFailure. This is how a human approval that takes two days fits in a Standard workflow. A Wait state that loops to check a table is polling.
• Heartbeats: a task timeout cannot tell a slow ten-hour job from a crashed worker. Set HeartbeatSeconds on the task and have the worker call SendTaskHeartbeat regularly; silence fails the task with States.HeartbeatTimeout. A Retry on that error runs the task again, sending a new message with a fresh token for another worker; a Catch to a Fail state would detect the crash but abandon the job. SendTaskSuccess cannot be used as a heartbeat because its first call completes the task.
• Saga compensation: when steps call separate systems (reserve stock, take payment, book a courier), a failure must undo what already succeeded. Add a Catch to each step that routes to compensating tasks (cancel the earlier bookings, in reverse order). Retries help transient errors but never undo partial work.

Distributed Map for very large datasets

An inline Map runs inside the parent execution, so it shares its limits (25,000 history events, a 256 KiB payload) and runs at most 40 iterations concurrently. A Distributed Map runs each item or batch as a child workflow execution with its own history, at up to thousands of concurrent children. Its ItemReader reads the dataset straight from S3 (objects under a prefix, a CSV or JSON file, or an S3 Inventory manifest); children can be Express (short idempotent items) or Standard. ToleratedFailurePercentage or ToleratedFailureCount lets the run continue past isolated failures (the default tolerance is zero), and a ResultWriter exports results to S3 instead of returning millions of outputs in the 256 KiB state output.

Lambda durable functions let you write a multi-step, long-running workflow as ordinary handler code in a supported runtime (such as Python or Node.js), using the durable execution SDK's steps and waits. Each completed step is checkpointed. A wait (for a duration or for an external callback) suspends the execution with no compute charge while waiting, and an execution can span up to one year, far beyond a single invocation's 15-minute limit.

Checkpoint and replay

When an execution resumes, Lambda replays the handler from the top: completed steps return their checkpointed results instead of running again, but code outside steps runs again on every replay. So the code must be deterministic: anything random or time-based (a UUID, the current time, a random sample) must be produced inside a step so its value is checkpointed. For example, a timestamp read outside any step gets a new value on each resume, so a log line or comparison built from it disagrees with what the earlier steps recorded. Invoke durable functions through a qualified ARN (version or alias) so that replays run the same code that started the execution. Pinning code and determinism are separate requirements: a pinned version that reads the clock outside a step is still non-deterministic, and no function setting changes the fact that replay re-runs code outside steps.

Rebuilding durability by hand (an SQS delay queue that re-enqueues the function, with progress in DynamoDB) is limited by the 15-minute maximum SQS delay and leaves you to write state and retry logic.

EventBridge routes events between accounts and Regions by having a rule in the sending bus target something in the receiving account or Region, and both sides must permit it.

Bus-to-bus routing across accounts

• Receiver: the destination bus needs a resource-based policy that allows events:PutEvents. A statement conditioned on aws:PrincipalOrgID authorises every current and future account in the organization, so the policy never needs editing when accounts are added; listing account IDs does.
• Sender: a rule in each source account targets the destination bus ARN with an IAM role that allows events:PutEvents on it. Rules that target an event bus in another account created since March 2023 require that role. Deploy the rule with a service-managed CloudFormation StackSet so new accounts get it automatically.
• A rule on the receiving bus can only match events already delivered to it; it cannot pull events from other accounts.

The one-hop limit

EventBridge does not forward an event received from another account on to a third account. If account A sends to hub account B and B's rule forwards to account C, B's own events reach C but A's never do, whatever bus type B uses, and replaying them from an archive meets the same limit. Route each source directly to every destination (a rule in A that targets C's bus, and A allowed in C's bus policy).

Across Regions

A rule can target an event bus in another Region (and in another account at the same time, in one hop, when the destination Region supports it); other target types, such as Lambda functions and SNS topics, must be in the rule's own Region. To process events from several Regions in one place, send them bus-to-bus into the destination Region and invoke the consumer from a rule on that bus. A bus only sees events published in its own Region. Routing through a Regional security bus and then on to a central Region fails because the second cross-account hop is dropped; target the central bus directly instead. (EventBridge global endpoints are a different feature: they fail event ingestion over to a secondary Region for resilience.)

Direct cross-account targets

Because EventBridge assumes the execution role in the bus account, that account's service control policies apply to the delivery. A role in the receiving account cannot be used by the sender's rule, and the bus's own resource policy controls who may put events on it, not where its events can go.

An Amazon API Gateway WebSocket API keeps a persistent, two-way connection to each client so the backend can push updates the moment they happen, instead of clients polling. Stage caching, short CloudFront TTLs and long polling all remain client-pulled.

• Routes: $connect runs when a client connects (store the connection ID), $disconnect when it disconnects, $default for messages that match no other route, plus custom routes chosen by a route selection expression on the message body.
• Pushing: the backend posts to a connection through the @connections management API (the connection's callback URL). A GoneException (HTTP 410) means the connection no longer exists.
• Connection store: Lambda functions share no memory, so connection IDs and subscriptions live in DynamoDB. For publish-subscribe fan-out, store each connection with its topics and add an index on the topic so the publisher can find exactly the subscribers of one symbol or room and post only to them.

Keeping the store clean: delete IDs in a $disconnect handler, and because $disconnect is best effort, also delete any ID that returns GoneException during a broadcast. Cleanup should happen on the two events that prove a connection is gone; time-based expiry or probing each connection first either lags behind reality or adds a call per client without removing anything.

Fixed connection limits: a WebSocket connection closes after 10 minutes idle and after a maximum of 2 hours, and neither quota can be raised. Clients therefore send a small heartbeat message at intervals under ten minutes and reconnect before two hours. Browsers cannot be SNS HTTPS subscribers, because SNS needs a public endpoint that confirms the subscription.

Tenant isolation is the set of controls that stop one SaaS tenant from reading, changing or starving another tenant's data and capacity. It is decided per tier, not once for the whole product.

Per-tenant encryption keys strengthen data isolation in a pool but do not un-share compute, so they do not satisfy a contract that forbids shared resources.

Noisy neighbours and tiered capacity

• SQS fair queues: setting MessageGroupId to the tenant ID on an existing standard queue marks each message's tenant; when one tenant builds a backlog, SQS favours other tenants' messages, with no new queue and no consumer change. On a standard queue the group ID does not order messages. Switching to FIFO or per-tenant queues adds queues, ordering constraints and consumer changes.
• API Gateway usage plans: a plan per tier with throttling and a quota, and an API key per tenant in its tier's plan, caps each tenant's request rate and monthly volume. API keys identify clients for metering; AWS advises against using them for authentication or authorization.
• Reserved concurrency for a premium tier: routing the premium tier to its own function with reserved concurrency sets capacity aside that pooled tenants cannot consume, without a stack per tenant. Provisioned concurrency on a function both tiers share is consumed by everyone and guarantees nothing to one tier.

In a pooled design, isolation is only as strong as the tenant context it trusts and the layer that enforces it. The goal is that a coding mistake in the application cannot return another tenant's data.

A trustworthy tenant claim from Amazon Cognito

Keep the tenant ID in a Cognito custom attribute that the app client can read but not write, and set it from the back end with AdminCreateUser or AdminUpdateUserAttributes. If the app client can write the attribute, a signed-in user can call UpdateUserAttributes with their own access token and switch tenants. Client metadata, request headers and browser storage are all user-controlled and cannot establish tenant context.

IAM-enforced isolation in DynamoDB

A single execution role that can read the whole table enforces nothing per tenant; a FilterExpression is application logic applied after items are read. Instead, for each request assume a scoped role with the tenant ID from the validated token as a session tag, and allow access only where dynamodb:LeadingKeys equals ${aws:PrincipalTag/tenant}. With partition keys that start with the tenant ID, IAM denies any other tenant's items. Account-level conditions such as aws:SourceAccount cannot tell tenants apart.

Row-level security in Aurora PostgreSQL

For shared tables with a tenant_id column, enable row-level security on each table with a policy such as tenant_id = current_setting('app.tenant_id'), and have the application set that value for each transaction from the verified token. PostgreSQL then filters every query, even one missing its WHERE clause. Who the policy does not bind matters: superusers and roles with BYPASSRLS always skip it, and the table owner skips it unless FORCE ROW LEVEL SECURITY is set, so the application must connect as a separate, non-owner role without those attributes. One policy per table serves every tenant; designs that create an object per tenant (a view, schema or database login each) also isolate rows, but their object count grows with the tenant list.

Tenant-specific encryption with AWS KMS

With one shared key, decryption must present the same encryption context used to encrypt. A key policy statement that allows kms:Decrypt only when kms:EncryptionContext:tenantId equals ${aws:PrincipalTag/tenantId} lets each tenant-tagged session decrypt only its own tenant's data. The general rule: a condition isolates tenants only if it compares the tenant value in the request with the tenant identity of the caller. Conditions on which context keys exist, on the key's own tags (one shared key carries one set of tags) or on which AWS service is calling are all true for every tenant alike.