AIP-C01 Application Integration Patterns and Developer Tools: API Gateway, Amplify, Bedrock Flows, Step Functions and Troubleshooting

A foundation model (FM) API interface is the HTTP front door that callers use to reach a model, and it has to cope with two things ordinary APIs rarely meet: answers that take tens of seconds or minutes to generate, and output that is most useful when it arrives token by token. In Amazon API Gateway the answer depends on the API type, because each type has different timeout and streaming behaviour.

Integration timeouts by API type

REST API response streaming

Setting an integration's response transfer mode to STREAM (the default is BUFFERED) makes API Gateway forward bytes to the client as the backend produces them. It works for HTTP_PROXY and AWS_PROXY (Lambda proxy) integrations on every REST endpoint type. Time to first byte drops to the time of the first token, and a streamed response can run for up to 15 minutes, which also gets past the 29-second integration timeout without a quota increase. The idle timeout between chunks is 5 minutes for Regional and private endpoints and 30 seconds for edge-optimized ones. The backend has to cooperate: a Lambda function must use a streaming handler that writes chunks as ConverseStream yields them.

Because API Gateway no longer holds the whole response, anything that needs the full payload stops working on a streamed method: endpoint caching, content encoding (compression) and VTL mapping templates on the integration response. If those behaviours must remain, they move into the backend: the function strips internal fields before writing each chunk, and it compresses the payload itself and sets the Content-Encoding header. As a rule, any feature that needs to see the whole payload before sending it has to move into the backend once the response streams.

Choosing between a longer timeout, streaming and an asynchronous job

• Raise the REST integration timeout when the client needs one complete response (for example it parses only finished JSON) and the backend code must not change. Only Regional and private REST APIs allow it. A longer Lambda timeout alone does nothing, because API Gateway closes the connection first.
• Stream when users should see text as it is generated and the backend can be changed to write chunks.
• Use asynchronous request-reply when the work outlasts what the client can hold open, or the client cannot read a stream. A POST starts the work (for example an AWS Step Functions execution) and returns 202 Accepted with a job ID at once; a GET status method returns the stored result when it is done. The client polls.

WebSocket APIs

A WebSocket route that collects the whole answer and returns it as the route response fails once generation passes 29 seconds, and there is no STREAM mode for WebSocket routes. The pattern that works is to acknowledge the message quickly, hand the work to a worker invoked asynchronously, and have the worker send each chunk to the client's connection with PostToConnection on the @connections endpoint as the model streams. Server-sent events, SSE versus WebSockets and SDK backoff in general are covered in the Task 2.4 lesson on FM API integrations; this task is about shaping the API layer itself.

Token limit management means controlling how many tokens enter and leave each model call, so requests fit the model's context window and spend stays predictable. It is enforced at three points: before the call, in the call's inference parameters, and at the API edge.

Before the call: CountTokens

The Amazon Bedrock CountTokens API takes the same body you would send to InvokeModel or Converse and returns the input token count for that specific model, which is the count Bedrock bills. It is free, so it suits pre-checks against the context window and cost estimates before a large request. Rules of thumb such as characters divided by four count the wrong unit for any given tokenizer, and a test inference with a tiny output limit is still billed for its input.

In the call: maxTokens and stopReason

maxTokens in the Converse inferenceConfig caps how many tokens the model may generate. When generation stops at the cap, the response's stopReason is max_tokens and the invocation log's output.outputTokenCount equals the limit, which is how truncated answers show up. Bedrock bills the output tokens actually generated, so raising the cap adds no charge for answers that finish earlier; it only removes the truncation. One caution: for some models (Anthropic Claude among them) the maxTokens value is reserved against the tokens-per-minute quota when the request starts, so an oversized cap can trigger throttling sooner. Temperature, streaming and Lambda timeouts do not change the token limit. stopSequences end output at a chosen string but do not guarantee a cap, so maxTokens is the control that bounds billed output.

At the edge: request validation

A REST API request validator with a JSON schema model can set maxLength on a text field, so API Gateway rejects an oversized body with a 400 response before the integration runs. That saves the Lambda invocation and the input-token charges that a check inside the function (even an accurate CountTokens check) would already have incurred. Usage plans limit request rates and counts per API key, not request size.

Model timeouts and retries

Long generations often fail with a botocore ReadTimeoutError after about a minute even though the Lambda function has minutes left: the SDK's default read timeout is 60 seconds. The fix is to create the bedrock-runtime client with a Config whose read_timeout exceeds the longest expected generation. More retries or adaptive retry mode only multiply the waiting.

Retries must fit the response budget: with a 25-second promise to users, the total of attempts and delays has to stay under it. Fixed delays and immediate retries synchronise many clients and deepen throttling; jitter spreads them out. API Gateway has no feature that retries an integration for you.

The AWS Amplify Gen 2 AI kit lets a web or mobile team add Amazon Bedrock features by declaring AI routes in the Amplify data schema and rendering them with ready-made React components and hooks, so very little backend or UI code is written. It offers two kinds of route, and choosing the right one is the main decision.

The UI pieces come from @aws-amplify/ui-react-ai, and createAIHooks generates the typed hooks for your schema. Amplify wires Amazon Cognito sign-in, AppSync, Lambda and Bedrock behind the routes; hand-building the same thing with AppSync resolvers, DynamoDB tables and a custom chat view works but is far more code.

Giving a conversation access to the user's data

A conversation route can carry tools the model calls during a chat. a.ai.dataTool pointed at an Amplify data model exposes it to the model; for a model, list is the only supported operation. Amplify runs the tool with the signed-in user's identity, so a model protected by owner-based authorization returns only that user's records. A shared knowledge base, or a Lambda function that reads the table with its own role, has no per-user scoping and bypasses those rules.

API-first development means the OpenAPI definition, kept in version control, is the source of truth and the API is created from it, rather than built in the console and documented afterwards. In API Gateway this works fully for REST APIs.

• Import an OpenAPI 2 or 3 file to create a REST API (import-rest-api) or update an existing one (put-rest-api). API Gateway extensions carry what plain OpenAPI cannot: x-amazon-apigateway-integration describes each backend, and x-amazon-apigateway-request-validators (with x-amazon-apigateway-request-validator per method) enforces the declared schemas before any model is called.
• Export a deployed stage with the API Gateway extensions to capture integrations and validators, so the file can recreate the API in another account. A plain or Postman-flavoured export, documentation parts or a generated SDK leave the integrations out.
• Generate client SDKs from a deployed stage for Java, JavaScript, Android, iOS (Objective-C or Swift) and Ruby, giving partner teams typed clients.

Updating from the file: merge or overwrite

put-rest-api runs in merge mode by default: resources in the file are added or updated, but resources missing from the file are left in place, so a route deleted from the file keeps answering. Overwrite mode makes the API match the file exactly while keeping the same API ID and stage URL. Either way, REST API changes reach clients only after a new deployment to the stage. Keeping the same API and redeploying is what preserves the clients' URL.

The HTTP API gap

HTTP APIs also import OpenAPI, but they have no request validation and no SDK generation, so a contract-first design that needs both belongs on a REST API.

Amazon Bedrock Flows (formerly Prompt Flows) is a visual builder that links prompts, knowledge bases, Lambda functions, agents and logic into a workflow that applications call through one API, so business analysts can build and change GenAI workflows without writing code.

Nodes that come up most

• Prompt nodes (inline or referencing a Prompt Management prompt), knowledge base, Lambda function and agent nodes do the work.
• A condition node branches on a value.
• An iterator node takes an array and sends its items one at a time to downstream nodes. Without anything after it, the output node returns a separate response per item.
• A collector node, placed after the per-item processing, gathers the iterated outputs (with the array size) into a single collectedArray, so the flow returns one response. Iterator before the per-item node, collector after it.

Releasing and rolling back

Edits happen in the flow's working draft. Publishing creates an immutable version, and an alias points at a version. Production applications call InvokeFlow with the flow and alias identifiers, so analysts can keep editing the draft without touching production. Promoting a release, or reverting a bad one, is an update to the alias's routing configuration; the application does not change or redeploy. Pinning older Prompt Management versions can roll back one prompt, but not a whole multi-node workflow. How flows govern prompts (conditions for prompt selection, approvals) is covered in the Task 1.6 lesson on prompt management.

A CRM enhancement adds FM output, such as a call summary or a handover note, to records in a customer relationship management system, usually from a Lambda function triggered by a business event. Three production concerns decide whether it works: duplicates, the CRM's rate limits and its credentials.

Duplicates: make the handler idempotent

Amazon EventBridge delivers events at least once, and Lambda can run an asynchronous event more than once even after a successful run. A function that writes to an external system must therefore be idempotent. The Powertools for AWS Lambda idempotency utility, keyed on a stable business ID (the call or deal ID) and backed by DynamoDB, returns the stored result when a completed event repeats, which avoids a second note and a second model charge. If the handler raises an error, the utility removes the in-progress record, so Lambda's normal retries still rerun genuinely failed work. The principle: make repeats harmless rather than trying to prevent them, because at-least-once delivery means a retry can always happen.

CRM rate limits: control concurrency with a queue

CRM credentials

To rotate a CRM client secret without redeploying, store it in AWS Secrets Manager with a rotation schedule and read it at runtime. At high invocation rates, read it through the AWS Parameters and Secrets Lambda Extension, which caches it locally and refreshes it after a TTL, rather than calling GetSecretValue on every invocation. The value that changes must live in a secret store the function reads at runtime, not in configuration baked into the deployment.

Automated document processing turns large volumes of files into structured data, and AWS offers two complementary tools for it: AWS Step Functions to orchestrate many model calls at scale, and Amazon Bedrock Data Automation (BDA) to extract content from documents, images, audio and video through one API.

Step Functions at scale: Distributed Map

A Distributed Map state can read items directly from an Amazon S3 prefix and run each in a child workflow execution, so the parent's execution history limit is never reached even with hundreds of thousands of documents. MaxConcurrency keeps parallelism under the account's Bedrock request quota, ToleratedFailurePercentage (or count) lets the run succeed while a small share of items fail, and results can be written to S3 for reporting. An Inline Map runs at most 40 concurrent iterations inside one execution's history and payload limits; a Parallel state has fixed branches; a single Lambda function stops at 15 minutes.

Large payloads: the Bedrock optimized integration

Step Functions limits state input and output to 256 KiB, in Standard and Express workflows alike, so passing a whole contract in the Body of a Bedrock task fails with States.DataLimitExceeded. The Bedrock InvokeModel optimized integration adds two fields that the Bedrock API itself does not have: Input (an S3Uri to read the request body from) and Output (an S3Uri to write the response to, leaving only a reference in the state). Body and Input are mutually exclusive.

Bedrock Data Automation

BDA returns two kinds of output:

• Standard output, with no blueprint, gives the default insights for each modality. For video that includes a full summary, chapter (scene) summaries, the audio transcript and text detected on screen; for documents, text, layout and tables. A project lets you turn individual outputs on or off.
• Custom output uses blueprints that define the fields to extract (with confidence scores). A project can hold up to 40 document blueprints, and BDA automatically matches each document to the right one.

With document splitting enabled, BDA first divides a multi-document file (a loan packet, say) into individual documents and then matches each to its blueprint, so no classification model has to be built. A single merged blueprint cannot separate the documents. Processing runs asynchronously with InvokeDataAutomationAsync and the project ARN, writing results to S3. BDA can also act as the parser for a Bedrock knowledge base, but a knowledge base answers questions; it does not return extracted fields. Chaining Amazon Textract, Amazon Transcribe or Amazon Rekognition with custom classification and a model works, but it is the high-code path BDA replaces.

Developer productivity for GenAI applications means using an AI coding assistant to write, refactor, test and review the code around the models, and testing that code in a way that is fast and repeatable while model quality is evaluated separately.

Amazon Q Developer in the IDE

• Chat with the open project as context: paste a stack trace from a failed Bedrock call and it explains likely causes (a missing inference profile, a malformed message list, a missing IAM permission) and proposes code changes.
• Inline code suggestions as you type, including for AWS SDK and Bedrock API calls.
• Agentic coding that edits files and shows the changes as reviewable diffs: refactoring duplicated code, generating unit tests and documentation.
• Code reviews on a whole project or on changes, covering static analysis (SAST), secrets detection (such as a hardcoded API key), infrastructure-as-code issues (such as an unencrypted bucket in a CloudFormation template), code quality, deployment risks and software composition analysis, with suggested fixes, before code is committed.

Older guidance names /dev, /test, /doc and /review commands; those names are retired, and the same capabilities are reached through chat and agentic coding. AWS has announced end of support for the Q Developer IDE plugins on 30 April 2027, with Kiro as the named successor; the exam guide still names Q Developer, so learn it and plan for that move. Amazon Q Business is a different product: an enterprise assistant over company content in a web experience, not an IDE coding tool. Amazon Inspector scans deployed workloads and connected source repositories, and Amazon Macie classifies data in S3; neither reviews the uncommitted code in your IDE, and CloudFormation Guard evaluates infrastructure templates against policy rules.

Testing GenAI components

Model output is not guaranteed to be identical between calls, even with sampling turned down, so exact-match assertions belong only on deterministic code paths. Calling the real model in unit tests makes CI slow, costly and flaky. Performance optimization belongs in the same loop: measure latency per step (see tracing below), then shorten prompts, cap output, stream, or move a step to a smaller model.

Advanced GenAI applications split work across several model calls and agents, and the skill is picking the pattern whose control flow matches the task: a fixed chain, a feedback loop, a router, a planner with workers, or a team of agents.

Prompt chaining

Prompt chaining decomposes a task into sequential, focused model calls, each consuming the previous output. Compared with one long prompt it gives an inspectable output per step, a fixed order, gates between steps (a Choice state or schema validation that ends the chain early when a step finds nothing or returns malformed JSON), and a model per step, so cheap steps such as extraction can run on a smaller model that meets the accuracy bar. Two ways to orchestrate a chain:

• Step Functions Task states calling Bedrock, with every step's input and output in the execution history. Developers own the definition.
• Bedrock Flows with linked prompt nodes, called through an alias with InvokeFlow. Non-developers can reorder and edit steps in the visual builder.

An agent is the wrong fit when the order must be fixed, because it decides its own next step; step-by-step instructions, larger maxTokens or prompt caching inside one prompt do not add gates or per-step visibility.

Agent design patterns in Step Functions

A Retry field handles task errors and reruns the same input, so it cannot carry feedback; Parallel branches are fixed at design time; Map iterations are independent of each other.

Standard or Express for agent workflows

Express workflows run for at most 5 minutes and do not support the .waitForTaskToken callback pattern. A loop that needs human approval lasting hours or days must run as a Standard workflow that pauses on a .waitForTaskToken task until the review app returns the token with the decision. The pattern to recognise is a callback: the workflow hands out a token and resumes only when something external returns it.

Strands Agents and AWS Agent Squad

Strands Agents is AWS's open-source SDK for building model-driven agents: the model plans and decides which tool to call in a loop that the SDK runs for you, tools are plain decorated functions or MCP servers, and it supports multi-agent patterns such as agents-as-tools, swarms and graphs. AWS Agent Squad (now maintained at 2fastlabs/agent-squad, formerly awslabs), by contrast, orchestrates several separately built agents. Agent Squad's classifier routes each message to the single best agent, which fails when one question needs several specialists. A SupervisorAgent uses an agent-as-tools design: a lead agent calls the specialist agents (in parallel where useful) with shared context and composes one answer. Registering the SupervisorAgent with the classifier beside the specialists keeps simple questions on their direct route while cross-domain questions go to the supervisor; making the supervisor the only route changes how every simple question is handled. Strands multi-agent patterns (swarm, graph, workflow) and agent loops in depth are in the Task 2.1 lesson on agentic AI.

Troubleshooting an FM application means finding out what was sent to the model, what came back, how long each hop took and what an error actually means. The AWS tools map to those questions: model invocation logs with CloudWatch Logs Insights, distributed tracing with X-Ray, and AI-assisted analysis with CloudWatch investigations and Amazon Q Developer.

Analysing prompts and responses with Logs Insights

Amazon Bedrock model invocation logging is off by default; once enabled it delivers to CloudWatch Logs, Amazon S3 or both. Each record includes identity.arn (the calling role or user), modelId, operation, input.inputTokenCount, output.outputTokenCount, any requestMetadata, and the request and response bodies. Ranking callers by token volume is a Logs Insights aggregation on those fields:

stats sum(output.outputTokenCount) as out, count(*) as calls
  by identity.arn, modelId

The by clause chooses what you rank (callers, models or both), and the aggregate chooses what you measure: sum() over a token field measures volume, while count(*) measures requests. Bodies are written inline only up to 100 KB; larger bodies and binary data such as images go to an S3 large-data delivery location, with a reference in the log record. With CloudWatch Logs as the only destination and no S3 location configured, those bodies are not captured at all; log retention and IAM permissions do not change this.

Tracing FM API calls

A per-request breakdown of where time goes (DynamoDB, a knowledge base Retrieve, the Converse call) needs distributed tracing: turn on tracing for the API Gateway stage and the Lambda function, and instrument the AWS SDK clients so each downstream call becomes a span (subsegment) on the trace timeline. AWS put the X-Ray SDKs and daemon into maintenance mode on 25 February 2026 (end of support 25 February 2027) and recommends OpenTelemetry-based instrumentation such as the AWS Distro for OpenTelemetry (ADOT) for new work; ADOT spans sent to AWS are stored as X-Ray trace data. CloudWatch Lambda Insights and API Gateway's integrationLatency are too coarse to split one request into its downstream calls.

• The trace map in the CloudWatch console draws the services in a request path as connected nodes with latency, error and fault rates, which shows where a slowdown starts.
• X-Ray's default sampling records the first request each second plus 5% of the rest, so rare slow requests are usually missed at higher traffic. A custom sampling rule for the service, with a higher fixed rate and a priority above the default rule, captures far more during an investigation. X-Ray Insights and Lambda Insights analyse data; they do not change sampling.

Recognising GenAI-specific error patterns

• Truncated answers: stopReason of max_tokens and an output token count equal to the cap.
• Bursts of throttling or model timeouts at peak: quota pressure, handled with backoff, queues or more capacity.
• ValidationException "input is too long": the context window; trim or summarise the history.
• ValidationException saying on-demand throughput isn't supported and to use an inference profile: some newer models can be invoked only through an inference profile. Use the cross-Region inference profile ID as the modelId; a geographic profile (for example EU or US) keeps processing in that geography at on-demand pricing, while a global profile can route to any supported Region. Provisioned Throughput changes the billing model, and IAM changes do not fix this error (the role does need permission on the profile and the models it routes to).

For AI-assisted diagnosis, CloudWatch investigations scans related metrics, logs, CloudTrail change events and traces and proposes root-cause hypotheses. After an investigation group is set up, a metric alarm can carry an investigation action that starts one automatically when the alarm enters ALARM, so the analysis is waiting for the on-call engineer with no custom code. Amazon Q Developer covers the other end: inside the IDE, it reads an error together with the project's code and proposes the fix.