AIP-C01 Selecting and Configuring Foundation Models on AWS

Choose a foundation model (FM) in two passes: first rule models in or out on hard capabilities and limitations, then compare the survivors on your own prompts. A public leaderboard rank or a low token price never rescues a model that cannot accept the input, cannot return the output in one response, or is about to be retired. Every Amazon Bedrock model has a model card (and a row in the supported-models tables) that lists the facts you filter on.

Hard filters from the model card

• Input and output modalities. A workload that sends photos, scanned documents or audio needs a model that accepts that modality. A text-only model is out, however well it ranks.
• Context window (maximum input tokens). The whole request has to fit: system instructions, retrieved or pasted documents, conversation history and the question. If a document must go in one request without chunking, the context window decides.
• Maximum output tokens. A separate limit from the context window. Translation, rewriting and long-form generation produce output about as long as the input, so a model with a large context window can still stop partway through a long answer. Setting maxTokens higher than the model's documented maximum does not raise it.
• Languages, features and API support: supported languages, tool use, system prompts, streaming, structured output, and whether the model works with the Converse API, batch inference, Knowledge Bases or Agents.
• Regional availability and inference options: on demand in your Region, only through a cross-Region inference profile, or with Provisioned Throughput.
• Lifecycle state: Active, Legacy or end of life (see the lifecycle section).

Telling the size limits apart

When you see a symptom, map it to the limit that produces exactly that symptom: truncation is an output limit, rejection of a large input is the context window, and errors only at peak are a quota.

Compare shortlisted models with Amazon Bedrock model evaluation jobs run over a prompt dataset taken from the real workload. Public benchmarks measure general skill on someone else's data; they cannot tell you which model follows your house format, your brand voice or your policy text. An evaluation job runs one model (or a response set you supply) over your prompts and reports metrics, so running one job per candidate gives a repeatable, side-by-side comparison with little or no custom code.

Practical limits that shape the plan

• A custom prompt dataset is a JSON Lines file in Amazon S3. For model-as-a-judge jobs it is capped at 1,000 prompts per job, so a larger question set is split across several jobs per candidate.
• Human evaluation is the slowest and most expensive option; plan rater time and keep the prompt set to what your raters can review well.
• Reference-based algorithmic metrics are meaningless without ground truth; a judge model or humans fill that gap.
• Fine-tuning every candidate before choosing, or comparing only latency, answers a different question. Evaluate first, customize the winner later if needed.

Building a full evaluation system for production quality monitoring belongs to the evaluation and testing domain; here the job is narrow: pick the model that fits the use case.

Every Bedrock model is in one of three lifecycle states, and the state is a selection criterion as hard as modality or context size. Active models are fully supported. A Legacy model has an announced end-of-life (EOL) date and serves only customers who already use it. After EOL the model is removed and requests to it fail.

What this means for selection

Do not adopt a Legacy model for a new application, even if it scored slightly higher: a new account cannot adopt it, cannot buy Provisioned Throughput for it, and would face a forced migration soon. A Legacy model is not a safe fallback either. Customized models share their base model's fate: once the base goes Legacy you cannot retrain on it, and the custom model is removed at EOL, so move a recurring fine-tuning cycle to an Active successor early and migrate once.

Retiring and replacing a model

1. Watch for Legacy notices (console, model card, AWS Health / email) and record the EOL date.
2. Shortlist Active replacements and run evaluation jobs on your own prompts, re-tuning prompts written for the old model.
3. Roll the new model ID out gradually through configuration (AWS AppConfig) with alarm-based rollback.
4. Finish well before EOL. Workloads that run only occasionally deserve special attention: access can lapse during an idle stretch long before the EOL date, so migrate them before their next scheduled run rather than waiting for it.

The EOL date is a hard stop for the model itself: no purchasing or routing option keeps a retired Bedrock model running, and proprietary model weights are never exported for you to host. The only durable answer is migrating to an Active model. The lifecycle documentation also describes an extended access stage late in the Legacy period, during which pricing can rise before EOL, which is one more reason to migrate early.

The Amazon Bedrock Converse API (Converse and ConverseStream) gives every supported model one request and response shape, so switching models becomes a change to the modelId parameter rather than a new code path. InvokeModel takes each provider's native JSON body, so supporting another provider means writing and maintaining another body format. Streaming (InvokeModelWithResponseStream), asynchronous invocation and inference profiles do not change the body format.

What Converse normalises

• messages (conversation turns with text, image and document content blocks) and a system field for instructions.
• inferenceConfig for the common parameters only: maxTokens, temperature, topP and stopSequences.
• toolConfig for tool (function) definitions, with tool-use and tool-result blocks in the messages.
• additionalModelRequestFields for model-specific parameters such as top_k, passed through to the provider.

Gotchas when you swap the model ID

• Features are per model. The Converse supported-features table lists, per model, whether it supports system prompts, tool use, streaming tool use, document chat and vision. A request that uses a feature the model lacks fails with a validation error. Check the table when adopting a model; if a model lacks system prompts, a common workaround is to move the instructions into the first user message.
• Model-specific fields go only to the model that accepts them. Putting top_k in inferenceConfig fails validation for every model, and sending it in additionalModelRequestFields to every model can cause validation errors on models that do not accept it. Keep per-model extra fields in configuration and attach them only for that model, so the request-building code stays shared.
• A feature is a property of the model itself. Request parameters and routing wrappers change how a call is sent, not what the model can do, so the fix for an unsupported feature is a different model or a workaround in the request content.

Keep the model ID, inference parameters and per-use-case routing in AWS AppConfig and read them at run time, so changing models is a configuration deployment that takes minutes, rolls out gradually and rolls back automatically. A model ID in a Lambda environment variable or in code needs a function update or a release, which is slow and has no built-in gradual exposure.

The AppConfig pieces

• Configuration profile: a freeform document (JSON or YAML) or feature flags, stored in the AppConfig hosted store or in sources such as S3 or Parameter Store.
• Validators on the profile, a JSON Schema or a Lambda function, reject malformed configuration (for example a mistyped model ID) before it is deployed.
• Deployment strategies control blast radius: linear or exponential growth, a growth factor, deployment duration and a bake time. AppConfig watches CloudWatch alarms attached to the environment and rolls back automatically if one fires during deployment or bake time. An all-at-once strategy with an alarm still exposes every target before rollback.
• Feature flags (multi-variant flags) can split a share of users between two models for an A/B comparison.

Reading configuration from Lambda

The AWS AppConfig Agent Lambda extension fetches and caches configuration and serves it to the function over a local HTTP endpoint, polling for new versions while the execution environment stays warm. It is tuned with environment variables, so no code changes are needed:

Near-twin alternatives

Put one stable API in front of every model provider, typically an Amazon API Gateway API backed by a Lambda adapter, so client applications call a single contract and the platform team can move a use case between Bedrock, a SageMaker AI endpoint or an external provider without client changes. Bedrock and SageMaker AI have different runtime APIs and payloads; if clients call providers directly, every switch is a client change.

• The adapter maps each request (by use case, tenant or route) to the provider and model named in configuration, translates the payload, and normalises the response.
• The mapping itself lives in AppConfig, read through the Lambda extension, so a provider switch is a configuration deployment, not an adapter redeployment.
• Credentials for third-party providers (API keys) live in AWS Secrets Manager, read by the adapter at run time. An encrypted environment variable is still visible to anyone who can read the function configuration, and changing it needs a function update.
• API Gateway adds authentication, usage plans and throttling per client in one place.

The test for any proposed abstraction is simple: does one component translate payloads between the client contract and each provider's API, and is the routing owned by the platform, not the client? DNS-level traffic splitting moves requests but leaves payload formats untouched, anything that hands provider choice to clients makes each switch a client change, and Bedrock-native constructs (such as inference profiles) only ever point at Bedrock models.

Cross-Region inference lets you call a Bedrock model through an inference profile that routes each request from your source Region to one of several destination Regions, which raises available capacity and gives access to models that are not offered on demand in your Region. You invoke it by passing the profile ID (or ARN) as the modelId. Some models can be called on demand only through a profile; calling them by base model ID returns a validation error saying on-demand throughput is not supported and an inference profile should be used.

Geographic or global profile

Data stays in AWS's network, and the call is logged by CloudTrail in the source Region, with the Region that served it recorded in the event. You do not need to enable destination Regions in your account for routing to work.

Permissions

• An identity policy that allows bedrock:InvokeModel on a geographic profile must also allow the foundation-model ARN in every Region the profile lists. Allowing only the profile, or only the local Region's model, leaves routed requests denied. A bedrock:InferenceProfileArn condition can limit those model ARNs to use through the profile, which keeps least privilege.
• A global profile needs three resources: the profile ARN, the foundation model in the source Region, and the Region-less foundation-model ARN (arn:aws:bedrock:::foundation-model/...), whose requests carry an unspecified aws:RequestedRegion.
• Service control policies that restrict aws:RequestedRegion must allow every destination Region of a geographic profile, and must allow unspecified for a global profile, or alternatively exempt Bedrock invocations made through specific inference profile ARNs (a condition on bedrock:InferenceProfileArn) so the Region restriction does not apply to them. An SCP deny cannot be overridden by any identity policy.

Profiles and Provisioned Throughput

Inference profiles, both system-defined and application inference profiles (which you create from a model or profile to add tags for cost tracking), do not support Provisioned Throughput. Provisioned Throughput is bought for a model in one Region and invoked through its own ARN. To combine a dedicated baseline with burst capacity, send the baseline to the Provisioned Throughput ARN and route overflow, such as throttled requests, through the cross-Region profile on demand in application logic.

Match the resilience pattern to the failure: retry brief, transient faults; use a circuit breaker to stop hammering a dependency that is down for minutes; degrade gracefully so users get something useful when no model answers; and deploy across Regions for Regional outages.

Circuit breaker in AWS Step Functions

Each execution checks the circuit status in a shared store, normally a DynamoDB item, before the model task, and records failures after it. The circuit has three states: closed (call the model normally), open (after repeated failures, skip the model and take a fallback path, chosen by a Choice state), and half-open (after a cool-down, let a trial call through; success closes the circuit). Storing an expiry timestamp or TTL with the open state makes recovery automatic, with no operator reset. The state must be shared: execution input is private to one run, so concurrent executions would never agree. A circuit breaker is the wrong tool for faults that clear in seconds, because it diverts work long after the burst ends.

Graceful degradation

Graceful degradation serves a reduced but useful response instead of an error. Layer the fallbacks: a smaller or alternative model with a tight timeout; then non-generative results (approved FAQ answers, plain keyword search, a live-agent handoff); and keep core features working when the AI enhancement is skipped. Respect freshness rules: a cached generated answer is only a valid fallback for content that does not go stale; anything personal or real-time must come from its live source or not be shown. Unbounded retries and asynchronous replies break latency budgets.

Cross-Region deployment of custom models

Bedrock cross-Region inference covers Bedrock models only. A custom model on a SageMaker AI endpoint needs its own deployment in a second Region, created by the same release pipeline so both Regions run the same approved version, with Route 53 health-checked failover (or latency) records in front of the Regional APIs. SageMaker AI reads the model artifact from S3 and the container image from Amazon ECR in the Region where the model is created, so replicate the artifact bucket (S3 replication) and the image repository (ECR replication) to the second Region and point the second model at those copies. Extra production variants or Availability Zones stay inside one Region.

Deploy a customized model where its form and traffic pattern fit: Amazon SageMaker AI endpoints give full control and always-on capacity (including many LoRA adapters on one endpoint), while Amazon Bedrock Custom Model Import serves supported open-weight models on demand through Bedrock APIs with no instances to manage.

Parameter-efficient adaptation

Low-rank adaptation (LoRA) freezes the base model's weights and trains small low-rank matrices, an adapter, that adjust its behaviour. Because each adapter is small, one copy of the base model can be loaded once and many adapters attached to it. Full fine-tuned checkpoints, quantized copies and distilled student models are each complete models, so serving many of them means many copies in GPU memory.

Adapter inference components on SageMaker AI

• Deploy the base model as an inference component on an endpoint, then each adapter as an adapter inference component that references the base component and a LoRA artifact in S3. Adapters share the base component's compute.
• Clients choose the adapter per request by passing its InferenceComponentName to InvokeEndpoint, so many light, intermittent tenants or business units share one GPU fleet.
• To release a retrained adapter, add it as a new adapter inference component and repoint that tenant's calls. The old component stays in place for a rollback in minutes, and the base and other adapters are untouched. The principle is isolation plus a rollback point: any change made in place, or made to a shared resource (the endpoint or the base component), either destroys the previous version or touches every tenant.
• The advantage depends on sharing one loaded base model and choosing the adapter per request. Designs that load full merged copies multiply GPU memory, and traffic-splitting features route by percentage, not by tenant.

Bedrock Custom Model Import

• Imports open-weight models of supported architectures (for example Llama, Mistral and others listed in the docs) that were fine-tuned elsewhere, such as in SageMaker AI training.
• Input is an S3 location holding Hugging Face format files: .safetensors weights, config.json and tokenizer files. GGUF files, model.tar.gz archives with inference scripts and custom containers are SageMaker AI patterns and are not accepted.
• The imported model is invoked through Bedrock runtime APIs on demand, billed for use, which suits sporadic traffic with the least operational overhead.
• Bedrock may remove an imported model's copies after a period of inactivity; the next call returns ModelNotReadyException while it is restored. If even the first call after a quiet period must be fast, a SageMaker AI real-time endpoint that stays running is the reliable choice, at always-on cost.

Other hosting options and where they fall short

Version customized models in the SageMaker Model Registry, release them through an automated pipeline triggered by approval, and roll out each release with SageMaker AI deployment guardrails that roll back automatically on CloudWatch alarms.

Model Registry and approval-driven pipelines

• A model package group holds the versions of one model; each model package version records its artifact, container and evaluation metrics.
• Each version has an approval status: PendingManualApproval, Approved or Rejected. A reviewer approves a version before production.
• A status change emits an Amazon EventBridge event, which can start the deployment pipeline (for example SageMaker Pipelines or CodePipeline) with no manual step.
• Rollback through the registry: if the pipeline deploys the latest Approved version, setting the bad version to Rejected makes the previous version the latest Approved one and the status-change event redeploys it, while stopping later runs from picking the bad version again. Manual changes made outside the pipeline do not change what the pipeline considers current, so they are undone the next time it deploys.

Deployment guardrails

All modes accept CloudWatch alarms in the auto-rollback configuration: if an alarm fires during the shift or baking period, SageMaker AI returns traffic to the old fleet. Blue/green always provisions a complete new fleet first; rolling updates provision only one batch at a time. A canary waits longer but still jumps from a small slice to everything, so it cannot reveal a fault that appears only at, say, half load; linear steps can.

Lifecycle management

Retiring a model is the last step of the same lifecycle: register the replacement, evaluate it against the incumbent, approve it, deploy it with guardrails, keep the previous version available for rollback, then delete the old endpoint or inference component and archive or reject the old registry version. For Bedrock-hosted models, follow the Legacy-to-EOL steps in the lifecycle section.