Deploying ML Models, Foundation Models and Agents on AWS (MLA-C02)

Choose the inference option from three facts in the scenario: whether a caller waits for the answer, how big and slow each request is, and whether traffic is steady, bursty or scheduled. Amazon SageMaker AI offers four options, and each one is the cheapest correct answer for exactly one traffic shape.

Two near-twins cause most mistakes. Serverless and asynchronous inference both cost nothing while idle, but only serverless answers synchronously, and only asynchronous accepts very large payloads, long run times and GPU instances. Asynchronous inference and batch transform both read from and write to Amazon S3, but asynchronous inference handles each request as it arrives (minutes of delay), while batch transform processes a fixed dataset as one job (results when the job ends). A real-time endpoint can always be made to work, so in a cost-constrained scenario it is usually the distractor that keeps paying for idle instances.

Asynchronous endpoints take the payload by reference and report completion through notifications; batch transform jobs are configured by how they split, batch and filter the input data.

Asynchronous inference

• The caller uploads the payload to Amazon S3 and calls InvokeEndpointAsync with its S3 URI as InputLocation. The call returns immediately with an OutputLocation where the result will be written. The payload is not sent in the request body.
• To be told when each request finishes without polling S3, set success and error Amazon SNS topics in the notification settings of the endpoint's AsyncInferenceConfig. Endpoint status events describe the endpoint, not individual requests, and data capture is a monitoring feature, not a result channel.
• An asynchronous endpoint keeps an internal queue, so it can scale to zero instances with an auto scaling policy whose minimum capacity is 0 and scale back out when requests arrive (the scaling policy itself is Task 3.2).

Batch transform data processing

• SplitType tells the job where record boundaries are (Line for CSV or JSON Lines). Without it, each file is sent as one request, which fails with payload-too-large errors for big files.
• BatchStrategy MultiRecord packs as many records as fit under MaxPayloadInMB into each request; SingleRecord sends one record per request. AssembleWith Line controls how outputs are joined back together, not how input is split.
• Instances divide the work by file (S3 object), not by line. One huge file is processed by one instance however many you add; split the data into several files to parallelize.
• InputFilter, JoinSource and OutputFilter (JSONPath expressions) run in that order: InputFilter chooses what the model receives (for example, with a record ID in the last column, $[:-2] sends every column except that ID), JoinSource Input appends the prediction to the original input record, and OutputFilter chooses what is written (for example $[-2,-1] keeps the ID and the appended prediction). Without JoinSource the output holds only predictions, so the ID cannot be kept.

Match the instance family to what the model actually needs: a CPU for small classical models, an inference accelerator for models that compile with the AWS Neuron SDK, and an NVIDIA GPU sized to the model's memory for everything else.

Right-sizing is a frequent theme: a model that needs one GPU with modest memory belongs on a single-GPU instance, not an eight-GPU one, and a small tree model gains nothing from any accelerator. Accelerators help only when the model's operators are supported by their software stack.

Use SageMaker AI hosting by default when the team wants AWS to run the serving fleet; choose a container orchestrator only when an organisational or technical constraint requires it. A SageMaker AI endpoint provides a managed HTTPS API, provisions and patches instances, runs health checks, replaces failed instances and scales with an auto scaling policy, with GPU and Inferentia instance types available.

When a scenario states that a small team has no container or Kubernetes experience, the managed endpoint wins; when it states that every service must use the existing Kubernetes tooling, EKS wins even though SageMaker AI would be less work in isolation.

All four put several models behind one endpoint to share instances, but they differ in whether models share a container, whether requests pass through one model or several, and whether each model gets its own resources and scaling.

Multi-model endpoint tuning

• Frequent model unloads and slow responses for regularly used models mean the cache is too small: choose an instance type with more memory so more models stay loaded.
• MMEs work best when models have similar traffic. A model with very high request rates or strict latency should move to its own dedicated endpoint so it stops evicting other models and gets consistent latency.
• GPU multi-model endpoints are served with the NVIDIA Triton Inference Server container; each model archive (model.tar.gz) follows Triton's model-repository layout with its config.pbtxt. One archive holding every model defeats on-demand loading.

Inference component scaling

Inference components scale in two layers. Each component's copy count scales with Application Auto Scaling and can go to zero so an idle model releases its accelerators. The number of instances behind the endpoint scales with managed instance scaling on the production variant (minimum and maximum instance counts): when new copies cannot be placed because every accelerator is allocated, SageMaker AI adds instances and removes them afterwards. Production variants are a different idea: each variant runs on its own instances and TargetVariant routes traffic between them, which is for A/B testing, not for packing models together.

A model fits on an instance only when its weights, plus room for the key-value (KV) cache that grows with batch size and context length, fit into the GPU memory it can actually use. Estimate weight memory as parameters × bytes per parameter: about 2 bytes in FP16/BF16, so a 7B model needs about 14 GB and a 34B model about 68 GB before any KV cache. Add headroom for the KV cache and runtime overhead before comparing with GPU memory.

• Tensor parallelism splits each layer's weights across several GPUs on one instance. In a SageMaker AI large model inference (LMI) container, set the tensor parallel degree (for example OPTION_TENSOR_PARALLEL_DEGREE) to the number of GPUs, otherwise the model loads onto one GPU and the others' memory is wasted. Combined GPU memory (GPU count × memory per GPU) is usable only when the model is sharded across all of those GPUs; for example, four 24 GB GPUs offer 96 GB to one model only at a tensor parallel degree of 4.
• Quantization shrinks weights when the instance is fixed: a 4-bit AWQ or GPTQ checkpoint needs roughly a quarter of the FP16 memory, at a small cost in quality. With vLLM in LMI, AWQ needs a pre-quantized checkpoint whose config the engine reads; on-the-fly options are formats such as FP8.
• Batching settings such as the maximum rolling batch size raise throughput but also raise KV cache memory; they never make an oversized model fit. Disk (EBS) is not a substitute for GPU memory.
• Cheapest instance that fits: compute total memory needed, find the smallest approved instance whose combined GPU memory covers it with headroom, then set tensor parallelism to use all of its GPUs.

On shared endpoints, resource allocation per model is declared on its inference component (for example the number of accelerator devices and minimum memory required), which is how several models share a multi-GPU instance without contending for the same GPU.

Pick a foundation model (FM) deployment option by who manages capacity and how you pay: per token with no infrastructure, a capacity commitment, a dedicated endpoint you size, or an asynchronous batch job.

The common traps: Provisioned Throughput is a commitment, so it is wrong when a scenario rules commitments out; batch inference cannot serve an interactive chat; and Bedrock batch inference is not SageMaker AI batch transform, which brings back instances and containers for a model you host yourself. Data-residency rules decide between geographic and global inference profiles.

Bring an externally trained model to AWS in one of two ways: import a supported LLM architecture into Amazon Bedrock with Custom Model Import, or host any model on SageMaker AI in a prebuilt or custom container.

Amazon Bedrock Custom Model Import

• Supports specific open model architectures (families such as Llama, Mistral, Mixtral, Flan-T5 and Qwen; check the current list). Upload the model in Hugging Face format to Amazon S3: weights as .safetensors plus config.json and tokenizer files. A raw PyTorch .pth checkpoint is not an accepted format.
• The imported model is invoked on demand through Bedrock APIs, with no endpoint to manage, billed for the compute units it uses while active.
• Limits to remember: embedding models are not supported, and imported models cannot be used with Bedrock batch inference. A model customization (fine-tuning) job is different: it retrains a Bedrock base model on your data rather than importing your weights.

SageMaker AI

• Prebuilt framework container (PyTorch, TensorFlow, scikit-learn, XGBoost, Hugging Face): package the weights and an inference.py with any custom loading or pre/post-processing (model_fn, input_fn, predict_fn, output_fn) in model.tar.gz, upload it to S3 and deploy. This is the least-effort path for standard frameworks.
• Custom container (bring your own): an image in Amazon ECR whose web server answers /invocations and /ping on port 8080. Use it for unusual runtimes or dependencies (building containers is Task 3.2).
• LMI or Triton containers for large language models and multi-framework GPU serving; embedding models that Bedrock cannot import are hosted here.
• The SageMaker Model Registry catalogs and approves model versions; it does not deploy anything by itself.

Deploy an agent either as a configured Amazon Bedrock Agents agent, where Bedrock runs the reasoning loop, or as your own agent code hosted on Amazon Bedrock AgentCore Runtime; then give it tools and connect it to other agents with standard protocols.

Action groups: how a Bedrock agent uses tools

• An action group defines operations with an OpenAPI schema or function details. A Lambda function executes the call, and the agent uses the live result in its reply. Knowledge bases only supply documents, guardrails only filter content, and text in the instructions cannot make network calls.
• Return of control (RETURN_CONTROL): the agent gathers the parameters and returns them in the InvokeAgent response; the application runs the action with its own context (session, fraud checks) and sends the result back in the session state. No Lambda function is needed.
• User confirmation on a function makes the agent present the exact proposed call for the user to confirm or deny before it runs; AWS recommends it as a safeguard against prompt injection. A prompt instruction to "ask first" is not enforced.
• To reach a private API (for example behind an internal load balancer), attach the action group's Lambda function to VPC subnets that can route to it. The agent itself never calls the API, so a VPC endpoint for Bedrock (which serves traffic into Bedrock) does not help.

Agent communication protocols

Agent memory and state management and the infrastructure behind agentic workflows are covered in Task 3.2.

Retrieval Augmented Generation (RAG) quality depends on retrieving the right chunks and passing only the best of them to the model; diagnose each symptom and change the one retrieval setting that addresses it. Creating knowledge bases and vector stores is Task 3.2; this task is about how queries are run.

Where reranking is configured

• With an Amazon Bedrock knowledge base, set a reranking configuration (a supported reranker model such as Amazon Rerank or Cohere Rerank) on Retrieve or RetrieveAndGenerate.
• For a custom pipeline that queries a vector store such as Amazon OpenSearch Service directly, call the Amazon Bedrock Rerank API with the candidate documents inline; no extra model has to be hosted. The knowledge base setting does not apply outside a knowledge base.

Retrieve versus RetrieveAndGenerate

Retrieve returns relevant chunks with metadata and source locations, so the application builds its own prompt, adds other data and calls any model. RetrieveAndGenerate (and its streaming form) retrieves and generates a cited answer with a chosen model, which is simplest but gives up control of the prompt. Search type (semantic versus hybrid) and chunking are covered with RAG optimisation in Domain 2.