AIP-C01 FM API Integrations: Bedrock Sync and Async Calls, Streaming, Resilience and Model Routing

Task 2.4 covers how application code talks to foundation models (FMs) reliably: calling Amazon Bedrock synchronously and asynchronously from any compute, streaming tokens to users in real time, surviving throttling and outages, and choosing which model serves each request. It has four official skills:

• 2.4.1 Flexible model interaction: Bedrock APIs called synchronously from Lambda, containers, Kubernetes or on-premises servers through the language SDKs; Amazon SQS for asynchronous processing; Amazon API Gateway as the custom API that validates requests and serves client SDKs.
• 2.4.2 Real-time interaction: Bedrock streaming APIs, WebSockets or server-sent events, and API Gateway response streaming (chunked transfer).
• 2.4.3 Resilience: SDK exponential backoff, API Gateway rate limiting, fallbacks for graceful degradation, and AWS X-Ray tracing across service boundaries.
• 2.4.4 Model routing: static routing in application code, Step Functions content-based routing, routing on observed metrics, and API Gateway request transformations.

Neighbouring topics own related material, so this lesson only points at them: agents and tool frameworks (Task 2.1), deployment choices such as Provisioned Throughput (Task 2.2), enterprise gateways and legacy integration (Task 2.3), and developer tooling (Task 2.5). Cross-Region inference, prompt routers and circuit breakers belong to the design topics in Domain 1. They appear in Task 2.4 scenarios mainly as alternatives that miss a stated requirement.

A synchronous call sends a request through an AWS SDK (bedrock-runtime client) and blocks until the whole answer is returned. Two things decide whether it works in production: the caller's credentials and the client's timeouts.

Converse or InvokeModel

Converse takes one request shape for every message-capable model (messages, system, inferenceConfig, toolConfig) and returns one response shape. InvokeModel takes each provider's native JSON body, so changing provider means changing code. Converse is what lets a model swap become a configuration change.

Credentials per compute environment

An AccessDeniedException from a container that otherwise works normally (pulls images, writes logs) usually means the permission sits on the wrong role.

Client timeouts

Long generations can outlast the SDK's defaults. In boto3 (Python), the socket read_timeout in botocore.config.Config defaults to about one minute, so a long report can fail with a read timeout even when the Lambda function still has minutes left. Raise the read timeout on the client. Raising the function timeout or the retry count doesn't help, because each retry hits the same limit and pays for the generation again.

The AWS SDK for Java 2.x separates the two budgets clearly:

• apiCallAttemptTimeout caps one HTTP attempt, so a slow attempt is abandoned and can be retried.
• apiCallTimeout caps the whole call, including every retry and backoff wait, so the service gets control back in time to serve a fallback.
• The HTTP client's connection timeout only limits opening the TCP connection. The socket timeout limits the gap between received bytes. Neither caps total duration.

Use asynchronous processing when nobody waits on the answer in the same request. Either buffer the work in Amazon SQS and let a consumer call the model at a controlled rate, or hand the job to Bedrock's own asynchronous APIs.

SQS → Lambda → Bedrock

• Cap the consumer with the event source mapping's maximum concurrency, not reserved concurrency. Maximum concurrency stops the poller from starting more invocations, so extra messages simply wait in the queue. With reserved concurrency alone, the poller keeps delivering batches, throttled invocations return messages to the queue, each attempt counts as a receive, and messages can reach the dead-letter queue (DLQ) without ever being processed.
• Visibility timeout of at least six times the function timeout (AWS's recommendation for SQS-triggered Lambda). If a long translation outlasts the visibility timeout, the message reappears and a second invocation processes it, so the model is billed twice. Long polling and FIFO deduplication IDs don't prevent this.
• Partial batch responses. Turn on ReportBatchItemFailures in the mapping's function response types and return failed message IDs in batchItemFailures. Only those messages return to the queue. Without it, one throttled call returns the whole batch and the successful items are sent to the model again.
• The redrive policy's maximum receive count decides when a repeatedly failing message moves to the DLQ. Batch size changes the number of messages per invocation, not how many invocations run at once.

Bedrock's own asynchronous paths

Calling StartAsyncInvoke once per record is not batch processing and gets no batch discount. A batch job per single user request misuses a bulk path. Provisioned Throughput (Task 2.2) is committed capacity for steady high volume and costs more than a batch job for a weekly run.

Put Amazon API Gateway in front of the model when clients need a stable, governed API instead of direct Bedrock access. Several of the features this skill names exist only on REST APIs, so the REST vs HTTP API choice decides the outcome.

Authorizers check identity, not request bodies. A Lambda authorizer doesn't receive the body on either API type, and a JWT authorizer checks tokens and scopes. Moving schema checks into an authorizer or into the function means custom code that a request validator makes unnecessary.

By default a validation failure returns a generic Invalid request body message. To tell callers which field failed without writing code, customize the BAD_REQUEST_BODY gateway response to include $context.error.validationErrorString. ($context.error.message only carries the generic text.)

When partners want ready-made client libraries but all traffic must still pass through your API, generate SDKs from the deployed REST API stage. Handing out the AWS SDK and IAM credentials bypasses the API entirely.

Streaming returns the answer as an event stream while the model generates it, so the first words appear in a second or two instead of after the full generation. ConverseStream streams in the unified Converse format. InvokeModelWithResponseStream streams each provider's native chunks. Shorter maxTokens, caching or latency-optimized inference may speed things up, but without streaming the user still sees nothing until the answer is complete.

The event sequence

1. messageStart: the role of the reply.
2. contentBlockStart: opens a block. For a tool call it carries the toolUse name and toolUseId.
3. contentBlockDelta: incremental content, either delta.text for text or a fragment of the tool input JSON string for a tool call.
4. contentBlockStop: the block is complete.
5. messageStop: carries stopReason (for example end_turn, max_tokens, tool_use).
6. metadata: carries usage (input, output and total tokens) and metrics (latency). Exact per-reply token counts come from here, not from messageStop, not from counting deltas (a delta isn't one token) and not from CountTokens, which counts input tokens only (no output tokens) and needs a separate call.

Streaming with tools

Tool input arrives as partial JSON strings spread across several deltas. Concatenate them until contentBlockStop and only then parse. When messageStop reports tool_use, the model is waiting for you: run the tool and send a toolResult block in the next request. A relay that only forwards delta.text drops tool calls silently.

Errors after HTTP 200

A stream returns HTTP 200 as soon as it opens. Later failures arrive as exception events inside the stream (internalServerException, modelStreamErrorException, serviceUnavailableException, throttlingException, validationException), and the SDK raises them while you iterate. Wrap the iteration loop too, not just the initial call, and start any fallback from there. SDK retries apply to the initial request only. A stream that has already delivered events is never replayed.

The delivery path must not buffer what Bedrock streams. Which path fits depends on whether the client only receives or also needs to send messages mid-answer.

A few details decide many real designs:

• Moving a streamed REST method off caching or VTL envelopes means doing that work in the backend: build the response envelope in the function, and cache behind the API. A non-proxy integration can't stream.
• CloudFront in front of a function URL can't stream a response that the function returns in one piece. Fix the source first.
• With WebSockets, a GoneException from PostToConnection means the client has left. End the ConverseStream loop right away, because output tokens are billed until generation stops. A $disconnect handler that only updates a table doesn't stop a generation already running in another invocation.

Exponential backoff with jitter is the AWS SDKs' built-in answer to transient failures. Each retry waits a random time up to a cap that grows with every attempt, so clients spread out instead of retrying in step. The SDKs offer three retry modes:

The adaptive rate limiter applies to the whole client instance. If one shared client serves many models, one throttled model slows every request it sends. Use standard mode there, or a separate adaptive client per model. Adaptive mode also delays first attempts, which hurts latency-sensitive paths. Default attempt counts differ by SDK and version, so set max_attempts explicitly instead of relying on a remembered number.

Custom retry loops (for example, around a non-SDK HTTP signer) need jitter too. A fixed doubling schedule, short fixed delays or a fixed start delay all keep clients synchronized, so throttles arrive in waves. Adding attempts on the same schedule extends the waves.

Retry in one layer. Retries multiply across layers: an app that retries 3 times, calling a function that loops 3 times around an SDK making 3 attempts, can send 27 or more model requests for one user action, right when the model is throttling. Keep the SDK's backoff as the single retry layer, remove extra loops, and have clients fail fast on 429 responses. Moving a loop into Step Functions Retry fields relocates the layer without removing it.

Which Bedrock errors to retry

Retry or fail over only on errors that can succeed later or elsewhere: throttling and server-side faults. A request that's wrong fails the same way on every attempt and on every model, so retrying it doubles the cost and still fails.

API Gateway rate limiting protects the shared model quota from any single caller and returns 429 Too Many Requests before a request reaches the backend. The limits sit in layers, and the exam scenarios turn on which layer is per-client:

A quota (requests per day, week or month) is not a rate: it doesn't stop a burst. Lambda reserved concurrency caps the function for everyone together and knows nothing about API keys. A Lambda authorizer that counts requests in DynamoDB can enforce per-key limits, but it adds custom code and a database call to every request, which usage plans already make unnecessary. API keys identify callers for usage plans. Pair them with a real authorizer for authentication.

Fallbacks within a latency budget

A fallback only helps if it gets time to run. With default retry settings, the SDK's throttling backoff can spend a whole response-time budget (say 5 seconds) before the code ever reaches the secondary model. Cap the attempts and per-attempt timeout used for the primary call so that its worst case leaves time for the fallback model to answer. Size the total retry budget to the response-time target, not the other way round.

When the AI output is optional (a personalised greeting on a dashboard, say), the cheapest graceful degradation is non-AI content: after a bounded retry, show static or cached text instead. Buying Provisioned Throughput (Task 2.2) or adding cross-Region capacity (Domain 1) can reduce throttling, but neither gives the page a path when the call still fails. A latency target needs a bounded retry and a fallback either way.

A distributed trace shows one request's path through API Gateway, Lambda and the Bedrock call, with each hop's duration, so you can tell whether the time goes to the API, your code or the model. Three pieces build it:

1. Enable X-Ray tracing on the API Gateway stage, so the trace starts at the API and context passes to Lambda.
2. Enable active tracing on the Lambda function, which records the function's segment.
3. Instrument the SDK client inside the function, so each Bedrock call appears as its own subsegment. AWS now recommends OpenTelemetry for this: the AWS Distro for OpenTelemetry (ADOT) or the AWS Lambda layer for OpenTelemetry, which auto-instruments the AWS SDK. The X-Ray SDKs and daemon are in maintenance mode (security fixes only) and are heading for end of support, so new instrumentation should use OpenTelemetry.

Annotations vs metadata. X-Ray indexes annotations, so filter expressions such as annotation.feature = "summaries" can search them. Metadata is stored but not searchable. OpenTelemetry span attributes become X-Ray metadata by default. To filter by one, list its key in aws.xray.annotations so the exporter writes it as an annotation. Sampling decides which traces are recorded, not which fields are indexed.

The other signals answer different questions. CloudTrail is an audit record of who called which API. Bedrock model invocation logs hold prompts and responses for the model call only. CloudWatch Logs Insights queries logs per log group. None of them joins the hops of one request with their timings.

Model routing decides which FM serves each request. Pick the simplest router that the request already allows: if the request type already names the model, a lookup table is enough.

Static routing that stays a table edit

Build the routing table on Converse, so the model ID is the only thing that changes between models. With InvokeModel (streaming or not), each new provider needs new body-building code. For per-feature cost reporting, create an application inference profile per feature from the model, tag it with the feature (an activated cost allocation tag), and put the profile's ARN in the table as the modelId. Each feature then gets its own costed resource even when two features share a base model. requestMetadata only labels model invocation logs for filtering. Bedrock can also allocate cost by IAM principal, but that can't separate features that share one execution role. System-defined cross-Region inference profiles can't be created per feature, though an application profile can be built on one.

Step Functions content-based routing

A Bedrock InvokeModel task (the optimized integration) classifies the request, and a Choice state branches on the result to a task that calls the specialist model. Model output is text: if the model replies with JSON, that JSON is a string inside the response body (for Anthropic Claude, Body.content[0].text). Parse it with States.StringToJson in a ResultSelector before the Choice state compares a field. Otherwise every execution falls to the Default branch. For short, high-volume routing that must answer in the same HTTP response, use an Express workflow started with StartSyncExecution (synchronous, billed per request and by duration). Standard workflows can't be started synchronously, bill per state transition and suit long-running, auditable processes.

Routing on metrics AWS already collects

Bedrock publishes metrics to CloudWatch in the AWS/Bedrock namespace by ModelId, including Invocations, InvocationLatency, InvocationThrottles, InvocationServerErrors, InvocationClientErrors and token counts such as OutputTokenCount. A scheduled function can read, for example, the p90 latency every few minutes and store the preferred model where the router reads it. To decide on failover, watch throttles and server errors. Ignore client errors: a malformed request would fail on any model, so counting them causes false failovers. Route 53 latency routing compares network latency to Regions, not inference latency between models.

Routing and transforming at the API edge

A REST API AWS service integration can call Bedrock directly with an IAM role. A path override (such as model/{modelId}/invoke) selects the model per method, and a VTL mapping template reshapes a simple client body into that model's request body. HTTP proxy integrations don't sign requests with SigV4 and pass bodies through unchanged. HTTP API parameter mapping can't rebuild a JSON body. When a template inserts user text into JSON, use $input.json('$.body'), which emits a correctly escaped JSON string. $util.escapeJavaScript turns an apostrophe into \', which is invalid JSON and fails with a ValidationException. URL or Base64 encoding produces valid JSON but sends the model the wrong text.