AIP-C01 Model Deployment Strategies: Bedrock Provisioned Throughput, SageMaker AI Endpoints and LLM Optimization

Task 2.2 is about choosing where and how a foundation model (FM) runs so that it meets an application's latency, throughput, cost and quality requirements. It has three skills: picking a deployment path (on-demand Amazon Bedrock calls from AWS Lambda, Bedrock Provisioned Throughput, or Amazon SageMaker AI endpoints, including hybrids of the two); solving the problems that make large language models (LLMs) different from traditional ML models (GPU memory, GPU utilization, token throughput and slow model loading); and right-sizing the model itself (smaller models for narrow tasks, cascades and routing).

Neighbouring tasks own related material: selecting and customizing models (including Custom Model Import, LoRA adapters and cross-Region inference profiles) is Domain 1; SDK retries, backoff and content-based routing with AWS Step Functions are Task 2.4; streaming APIs to clients are Tasks 2.4 and 2.5; cost and latency monitoring is Domain 4. Training and advanced data science are out of scope for the exam: you deploy and serve models, you do not train them.

Start from the traffic shape and the model you need. If a Bedrock model meets the quality bar and traffic is irregular, call it on demand: you pay per token, there are no servers or capacity to manage, and an idle day costs nothing. If the load is steady and must have reserved, predictable capacity, buy Provisioned Throughput. If the model is not served by Bedrock (an open-weight model you host yourself, a specialised model, or one needing a specific GPU setup), deploy it to a SageMaker AI endpoint.

Lambda for on-demand invocation

A Lambda function that calls Bedrock (InvokeModel or Converse) or SageMaker AI (InvokeEndpoint) is the simplest serverless wrapper. The common failure is the function's timeout: a new function defaults to 3 seconds, while a long generation can take tens of seconds or minutes. Set the timeout to cover the longest expected generation (the maximum is 15 minutes). Adding memory only adds local CPU; it does not make a remote model generate faster, and response streaming changes how tokens arrive, not how long the whole generation takes.

When Lambda calls a SageMaker AI endpoint, the InvokeEndpoint request must name the target that matches the hosting pattern: InferenceComponentName for endpoints that host inference components, TargetModel for multi-model endpoints, TargetVariant for a specific production variant, and TargetContainerHostname for multi-container endpoints. Moving a model into an inference component without adding InferenceComponentName makes existing calls fail validation.

Provisioned Throughput reserves dedicated model capacity in Bedrock at a fixed hourly price. In the model-unit mode you buy a number of model units (MUs), each delivering a set number of input and output tokens per minute for that model, and choose a commitment term. Billing runs every hour until the Provisioned Throughput is deleted, whether or not it is used, and requests above the purchased capacity are throttled.

Buying and using it

• Quota first. Model units must be granted to the account through an AWS Support quota case before purchase, or the purchase fails. AWS requires this for committed purchases and documents it for no-commitment purchases too.
• Units are fixed. For Provisioned Throughput by model units, you can edit only the name, tags and (for a custom model) the associated model. To change capacity you buy another Provisioned Throughput. A newer Provisioned Throughput by tokens mode, where you set input and output tokens per minute, does allow changing those token rates later.
• Route requests to it with the ARN. The purchase returns a provisioned model ARN. Put that ARN in modelId for InvokeModel, InvokeModelWithResponseStream, Converse or ConverseStream. A request that names the base model ID, or a cross-Region inference profile, runs on shared on-demand capacity instead.
• Agents use the alias. A Bedrock agent uses Provisioned Throughput when its alias is created or updated with routingConfiguration naming the provisioned throughput. Updating the alias that callers already use keeps the agent and alias IDs unchanged.

Custom models and swapping models

Fine-tuned Bedrock custom models are commonly served through Provisioned Throughput. For a Provisioned Throughput by model units, a base-model association can never change, but a custom-model association can be updated to its base model or to another custom model derived from the same base model. The provisioned ARN and the commitment stay the same, traffic keeps flowing during the update (some responses may still come from the old model until it completes), so callers need no change and no second purchase is needed. A committed Provisioned Throughput cannot be deleted early to buy a new one.

Baseline plus overflow

For steady load with occasional spikes, size Provisioned Throughput for the baseline and send requests that are throttled on the provisioned model to the on-demand model ID, as long as on-demand quota can absorb the spike. Sizing for peak pays for idle capacity most of the time, and retrying throttled calls against the same saturated capacity adds no throughput during a long spike.

Amazon Bedrock Marketplace is the main hybrid pattern: a Marketplace model runs on a SageMaker AI-managed endpoint whose instance type and count you choose, but you invoke it through Bedrock APIs. You call InvokeModel (or Converse, for models that support it) with the endpoint ARN as the model ID, and the model can be used with Bedrock tools such as Agents, Knowledge Bases and Guardrails (through the ApplyGuardrail API, for example). That keeps one code path and one set of safety controls across serverless Bedrock models and specialised Marketplace models.

The bring-your-own-endpoint rules matter when you have customised a model: a fine-tuned copy of a Marketplace model fails registration because its artifacts no longer match the provider's package. Another common hybrid splits a workload by model: a small in-house CPU model (a classifier, for example) on SageMaker AI Serverless Inference, with the large generative step on Bedrock on demand, so nothing is billed while idle and there are no instances to manage.

For LLMs, use real-time endpoints (preferably with inference components) for interactive traffic and asynchronous inference for large or slow jobs. Serverless Inference does not fit an LLM that needs a GPU: it offers no GPUs and at most 6 GB of memory (with a maximum concurrency of 200 per endpoint), so it suits only small CPU models.

Inference components

Inference components let one real-time endpoint host several models, each with its own resource needs, copy count and auto scaling. You declare each model's needs in its ComputeResourceRequirements (for example NumberOfAcceleratorDevicesRequired for GPUs per copy, plus CPU and memory), and the number of copies in RuntimeConfig.CopyCount. SageMaker AI places and packs copies onto instances. This is different from multi-model endpoints (which load models from S3 into shared memory on demand, typically for many small models), production variants (traffic splitting between variants) and inference pipelines (models chained in sequence).

Two layers scale independently: the copy count of each component (scalable dimension sagemaker:inference-component:DesiredCopyCount) and the endpoint's instances, which managed instance scaling on the production variant adds when copies need room and removes when they are idle (set MinInstanceCount and MaxInstanceCount).

Scaling a real-time endpoint to zero

• The endpoint must host inference components, and the variant's managed instance scaling must have MinInstanceCount 0 (with the component allowed to scale to zero copies). InitialInstanceCount itself cannot be 0.
• Scaling back out from zero needs a step scaling policy invoked by a CloudWatch alarm on NoCapacityInvocationFailures, the metric emitted when a request finds no capacity. Target tracking on a utilization metric cannot restart from zero because there is nothing to measure.
• Coming back from zero takes minutes (instance launch plus model loading), and requests fail until capacity is ready. When the busy hours are predictable, add Application Auto Scaling scheduled actions that raise the minimum copy count before the working day and lower it to zero afterwards, so users never meet the cold start and nights and weekends cost nothing.

Asynchronous inference

Asynchronous endpoints read each request payload from Amazon S3 (up to 1 GB), queue it, process it for up to one hour and write the result to S3, optionally notifying through Amazon SNS. They can scale to zero with a scalable target MinCapacity of 0. A target tracking policy on ApproximateBacklogSizePerInstance only scales out once the backlog passes its target, so a single queued request could wait; add a step scaling policy invoked by an alarm on HasBacklogWithoutCapacity, which signals queued requests with zero instances, to start the first instance.

An LLM's GPU memory is roughly its weights plus the key-value (KV) cache plus runtime overhead. Weight memory is the parameter count times bytes per parameter: 4 bytes in 32-bit, 2 bytes in 16-bit (FP16 or BF16), 1 byte in 8-bit and about half a byte in 4-bit. An 8B model in 16-bit therefore needs about 16 GB for weights, a 13B model about 26 GB and a 70B model about 140 GB.

The KV cache holds attention state for every token of every sequence being generated, so it grows with context length multiplied by the number of concurrent sequences. A model that runs fine with short prompts can run out of memory after the maximum context or the rolling batch size is raised. The fixes are to hold fewer long sequences at once (lower the maximum batch size), to give each replica more GPU memory, or to shrink the weights by quantization if precision may change. Adding instances adds replicas, not memory per replica, and a bigger EBS volume (VolumeSizeInGB) adds disk, not GPU memory.

Tensor parallelism

When a model does not fit on one GPU, tensor parallelism shards each layer's weights across the GPUs of one instance. In SageMaker AI Large Model Inference (LMI) containers you set it with option.tensor_parallel_degree in serving.properties (or the matching environment variable). If it is left at 1 or set too low, the whole model lands on GPU 0 and fails with out-of-memory errors while the other GPUs stay empty; set the degree so the shards fit, often equal to the instance's GPU count. Raising the instance count does not shard a model; each instance loads its own full copy.

LLM serving is limited by GPU memory and concurrency, not CPU, so throughput comes from how the container schedules token generation. With static batching, a batch keeps its slots until the longest response finishes, short requests leave the GPU idle and new requests queue. Continuous (rolling) batching admits new requests at each generation step and frees slots as sequences finish, raising throughput on the same hardware. In LMI containers you enable it with option.rolling_batch (for example vllm, or another supported engine) and cap it with option.max_rolling_batch_size, which also bounds KV-cache use.

Response streaming improves perceived latency for one user but does not change throughput. Larger or more instances add cost without fixing scheduling waste.

Auto scaling metrics for LLMs

Large model artifacts need specialised loading: avoid compressed archives, allow enough startup time, and use prepared artifacts that skip work at startup. A single model.tar.gz of tens or hundreds of gigabytes must be downloaded and extracted on every instance, which can outlast the deployment. Store the weights uncompressed under an S3 prefix and reference them with ModelDataSource and CompressionType set to None. Raise the production variant's ModelDataDownloadTimeoutInSeconds and ContainerStartupHealthCheckTimeoutInSeconds (each up to 3,600 seconds) so the container has time to load weights onto the GPUs before its health check fails.

Faster scale-out

For LLMs, scale-out time is dominated by what each new instance does before it can serve: download weights to disk, load them into host memory, shard them and copy them to the GPUs, and for some engines build a compiled engine. SageMaker AI inference optimization jobs remove those steps ahead of time:

• Fast model loading pre-shards the model and stores weights in chunks so that they stream from Amazon S3 directly to the GPUs, skipping the disk and host-memory steps. Accuracy is unchanged.
• Compilation produces hardware-optimized artifacts ahead of time: TensorRT-LLM for NVIDIA GPUs, and the AWS Neuron SDK for AWS Inferentia and Trainium. New instances no longer compile at startup, which cuts deployment and auto scaling time with no accuracy loss.

Moving a model from GPUs to Inferentia2 (ml.inf2) needs both Neuron-compiled artifacts and a container with the Neuron runtime, such as the Neuron variant of the LMI container. CUDA images and TensorRT-LLM engines are tied to NVIDIA GPUs, and quantization is optional rather than required. A longer health-check timeout only tolerates a slow start; it does not shorten it.

Match the optimization to the bottleneck and to any accuracy constraint: quantization saves memory but may cost accuracy, while speculative decoding, compilation, fast model loading and continuous batching keep the model's outputs as they are.

For speculative decoding, SageMaker AI optimization jobs offer a pre-built draft model, or you can supply your own; a team without a draft model or data scientists can start with the provided one. Speculative decoding speeds a single answer, while batching and scaling raise throughput across users without making one answer faster. Speculative decoding also adds the draft model's memory, so it does not help a model that does not fit.

The cheapest deployment is often a smaller model: if an evaluation on representative labelled data shows a smaller pre-trained model meets the quality bar for a narrow task (classification, extraction, routine questions), move that task to it and keep the larger model only where the evaluation says it is needed. Decide per task, not per application: a claims app may move classification and extraction to a small model while keeping letter drafting on a large one. When choosing among measured options, pick the cheapest model that meets every stated requirement (accuracy floor and latency target); Bedrock batch inference is cheaper still but asynchronous, so it cannot serve a user waiting on a request.

API-based model cascading

A cascade sends each request to a cheap model first and escalates to a larger model only when an automated check on the response fails. Because most traffic is routine, most requests never reach the expensive model. Escalation signals must be evidence that the cheap answer is wrong and must be available before the answer is delivered: validation against the expected JSON schema, a Bedrock Guardrails contextual grounding score below a threshold for answers grounded in retrieved documents, a self-reported low confidence, or a classifier. User ratings arrive too late, random sampling ignores quality and latency does not measure correctness. Running both models in parallel doubles cost.

Bedrock intelligent prompt routing

A Bedrock prompt router routes each request between two models from the same family (for example Amazon Nova Lite and Nova Pro) with no routing code. It predicts the response quality of each model per request and uses the non-fallback model only when its predicted quality beats the fallback model's by more than the configured response quality difference. With the cheaper model as fallback, a larger quality difference sends fewer requests to the expensive model; a smaller one sends more. You call it by passing the router ARN as modelId, and each response's trace.promptRouter.invokedModelId reports which model answered, which is how you reconcile cost per request.

Choose the router when its constraints match (same family, English, no code); build the cascade yourself for models from different providers, multilingual traffic, or escalation based on your own checks. Content-based routing with Step Functions and metric-based routing are covered in Task 2.4.