Provisioning ML Resources: Endpoints, Scaling, Knowledge Bases and Agents (MLA-C02)

Use on-demand capacity when traffic is unpredictable, bursty or idle for long stretches and you will not commit to a term; use provisioned capacity when traffic is steady and high, every request needs consistent latency, or the workload needs capacity that nothing else can consume. Skill 3.2.1 asks you to make this trade-off for both Amazon Bedrock and SageMaker AI, and the scenario's constraints (commitment, idle cost, cold starts, interactivity) decide it.

Amazon Bedrock

Only Provisioned Throughput reserves model capacity for one workload, and you must send requests to the provisioned model's ARN (not the base model ID) to use it.

SageMaker AI

A real-time endpoint runs instances you choose and bills for every hour they run, idle or not. Serverless Inference bills per request and memory size, runs on CPU only (no GPUs) and can cold start after idle time. Provisioned concurrency keeps a set number of serverless workers initialised, so requests avoid the cold start, and Application Auto Scaling can manage it on a schedule or with target tracking on the predefined SageMakerVariantProvisionedConcurrencyUtilization metric. A provisioned-concurrency scalable target has a minimum of 1, not 0: to cut idle cost outside business hours, schedule it down to 1 rather than removing it. Asynchronous inference can scale its instances to zero, but callers fetch results from Amazon S3, so it cannot answer interactively. Choosing the inference type itself is Task 3.1; Task 3.2 is about sizing and paying for the capacity behind it.

Pass values between AWS CloudFormation stacks with exported Outputs and Fn::ImportValue, or with AWS Systems Manager Parameter Store dynamic references when the producer must stay free to change the value; and drive SageMaker AI from AWS Step Functions with service integrations instead of custom polling code. Skill 3.2.2 tests which mechanism carries the value and which integration pattern waits correctly.

Step Functions integration patterns for SageMaker AI

Step Functions offers three patterns: request-response (call the API and move on), run a job (.sync, wait until the job finishes) and wait for a callback (.waitForTaskToken). The optimized SageMaker AI integration supports .sync only for jobs: training, processing, transform, hyperparameter tuning and labeling. It does not support .waitForTaskToken at all. CreateModel, CreateEndpointConfig, CreateEndpoint and UpdateEndpoint are request-response, so the state finishes as soon as SageMaker AI accepts the request, while the endpoint is still Creating.

To wait for an endpoint without AWS Lambda, build a loop: a Wait state, an AWS SDK integration task that calls DescribeEndpoint, and a Choice state that continues on InService, goes to a Fail state on Failed, and otherwise returns to the Wait. The principle: branch on the status the service reports, and treat a failed state as terminal.

Start from an AWS prebuilt image whenever you can, add only what it lacks, and store the result in Amazon ECR; a container that SageMaker AI runs must follow its fixed contract for ports and paths. Skill 3.2.3 covers the contract, the ways to customise images and how to keep repositories clean and scanned.

The inference container contract

SageMaker AI starts a hosting container and sends GET /ping health checks and POST /invocations inference requests to port 8080. A server on another port or with its own routes (for example /predict or /health) is never reached, so the endpoint fails its health checks and never becomes InService. The contract is all-or-nothing: the routes and the port must both match. Model artifacts from model.tar.gz are extracted to /opt/ml/model in the container.

The training container paths

A model saved to /tmp or any other path is discarded with the container, even though the job reports Completed. No AWS SDK upload code is needed.

Customising a prebuilt image

• requirements.txt beside the inference script: the prebuilt framework containers pip-install it when the container starts. It adds Python packages only (no OS packages), and it needs to reach a package repository at startup: the internet, or a private mirror such as AWS CodeArtifact through a VPC endpoint.
• Extend the AWS image: a Dockerfile FROM the AWS Deep Learning Container or framework inference image that installs OS and Python packages, pushed to ECR. Everything is baked in, and you keep AWS's serving stack and its patched base.
• Build from scratch (for example from a plain CUDA image): works, but your team now maintains the model server and every patch.

Maintaining images in Amazon ECR

Put the subnets and security groups in the VpcConfig of the SageMaker AI model (CreateModel), not in the endpoint configuration, and then give those private subnets a private path to every AWS service the model and its callers need. Skill 3.2.4 tests which VPC endpoint carries which traffic.

The model's network interfaces live in your subnets. With no NAT gateway, the endpoint cannot even download its artifacts from Amazon S3 until the subnets' route tables have an S3 gateway endpoint. Anything the inference code calls (Amazon DynamoDB, Amazon Bedrock, others) also needs an endpoint.

Three rules decide most questions. First, only Amazon S3 and Amazon DynamoDB offer gateway endpoints; every other service, including SageMaker AI and Amazon Bedrock, uses an interface endpoint (AWS PrivateLink). Second, runtime and control-plane APIs are separate endpoint services: an endpoint for bedrock or sagemaker.api does not carry bedrock-runtime or sagemaker.runtime calls. Third, code that uses the AWS SDK's default hostnames reaches an interface endpoint only when private DNS is enabled on it, so those names resolve to the endpoint's private IP addresses; and the endpoint's security group must allow HTTPS (port 443) from the caller's security group.

Network isolation

EnableNetworkIsolation is a CreateModel parameter that removes all network access from the container, including calls to AWS services, whatever the code inside tries. SageMaker AI itself still pulls the image and model artifacts, so the endpoint starts normally. It is the control for untrusted or vendor code. Security groups and route changes only filter traffic inside the VPC: they can block the artifact download or leave VPC endpoints reachable, so they do not prove the container is silent. Network isolation is not a way to reach S3 privately; that is the gateway endpoint's job.

Deploying with Boto3 or the AWS CLI takes three calls in order: create_model, then create_endpoint_config, then create_endpoint. Skill 3.2.5 tests the order, what each resource holds, how to swap a model without downtime and how to wait for an endpoint.

create_endpoint takes an endpoint configuration name, not an instance type. Endpoint configurations are immutable: to move an endpoint to a new model version, create a new model and a new endpoint configuration, then call update_endpoint with the new configuration. SageMaker AI provisions the new fleet, shifts traffic and keeps the same endpoint name in service throughout. Deleting and recreating the endpoint causes downtime, and create_endpoint with an existing name fails.

Creation is asynchronous: create_endpoint returns while the endpoint is Creating. To block until it can serve traffic, use the SageMaker AI client's endpoint_in_service waiter with the endpoint name (CLI: aws sagemaker wait endpoint-in-service --endpoint-name ...). The waiter polls DescribeEndpoint and stops on InService or errors on Failed. The waiter needs the name of the resource whose status it reads, which is the endpoint.

The SageMaker Python SDK wraps the same three calls: model.deploy(instance_type=..., initial_instance_count=...) creates the model, configuration and endpoint and returns a predictor, waiting for the endpoint by default. Infrastructure-as-code tools create the same three resources.

Track the metric that rises and falls in proportion to the load each instance or copy carries: invocations per instance for short, uniform requests; concurrent requests for long streaming requests; backlog per instance for asynchronous endpoints; and a resource metric such as GPU utilisation when requests differ widely in cost. Skill 3.2.6 tests this choice and the policies that make scale-from-zero work.

SageMaker AI scaling runs through Application Auto Scaling. The scalable dimension is the variant's instance count, an inference component's copy count, or a serverless variant's provisioned concurrency. Target tracking accepts predefined metrics directly; anything else needs a customized metric specification.

Mind the CloudWatch namespace in a customized metric. Invocation metrics (Invocations, ModelLatency, concurrency) are in AWS/SageMaker; instance metrics (CPUUtilization, MemoryUtilization, GPUUtilization, GPUMemoryUtilization) are in /aws/sagemaker/Endpoints. A specification with the right metric name in the wrong namespace finds no data. Endpoint-wide totals such as Invocations do not change with capacity, so they make poor target tracking metrics, and latency is an outcome rather than a load signal.

Scaling to and from zero

Target tracking cannot scale out from zero, because with no capacity there is no load metric to track. Each mechanism has its own wake-up signal:

• Inference components (real-time): set MinInstanceCount to 0 in managed instance scaling, give each component a scalable target with minimum 0 and target tracking, and add a step scaling policy driven by a CloudWatch alarm on NoCapacityInvocationFailures, which adds a copy when a request finds no capacity. Requests that arrive while the first copy starts are rejected. Real-time scale-to-zero is available only for endpoints that host inference components.
• Asynchronous endpoints: scalable target minimum 0, target tracking on backlog per instance, plus a step scaling policy on HasBacklogWithoutCapacity; queued requests wait rather than fail.
• Serverless endpoints scale to zero by design (no GPUs); provisioned concurrency floors at 1.

Scale GPU inference with inference components that declare each model's accelerator and memory needs plus managed instance scaling on the variant, and secure scarce GPU capacity for training with SageMaker AI training plans. Skill 3.2.10 tests both halves: packing and scaling models on GPU instances, and getting the GPUs at all.

Inference components

An inference component deploys one model onto an endpoint with its own compute requirements: number of accelerator devices, minimum memory and CPU cores. Each component scales its copy count independently with its own traffic, and SageMaker AI places copies on instances with free GPUs. Managed instance scaling on the production variant (minimum and maximum instance counts) adds instances when new copies do not fit and removes instances that are no longer needed. This is how several fine-tuned models share multi-GPU instances, each with dedicated GPUs.

For GPU workloads whose request cost varies, scale on GPUUtilization or concurrency rather than raw invocations (see the metrics section).

Getting GPU capacity

Read the error before choosing a fix. A service quota error means the account is not allowed more instances; an insufficient capacity error means AWS has none to give right now, and a quota increase alone does nothing. A SageMaker AI training plan reserves a chosen accelerated instance type and count for a chosen time window, for one target resource chosen at purchase: training jobs, SageMaker HyperPod clusters, SageMaker AI inference endpoints or SageMaker Studio apps. The target cannot be switched later, and the account still needs service quotas that cover the plan's instance count. Managed Spot training is cheaper but can be interrupted, and a SageMaker Savings Plan lowers the price of usage without reserving any capacity. Purchasing options and cost tooling in general belong to Task 4.2.

An Amazon Bedrock knowledge base turns documents into searchable vectors: it parses each source file, splits it into chunks, embeds each chunk with an embedding model and writes the vectors with their text and metadata to a vector store you choose. Skill 3.2.7 tests the vector store choice, the index settings that must match the embedding model, and the parsing, chunking and metadata settings that control what is indexed.

Choosing the vector store

Hybrid search combines keyword and semantic matching, which matters when users search for exact identifiers (part numbers, docket numbers, error codes) that embeddings represent poorly. Bedrock supports it only on Amazon RDS/Aurora, Amazon OpenSearch Serverless and MongoDB with a filterable text field; on other stores a HYBRID query falls back to semantic search.

Aurora prerequisites: Amazon Bedrock does not open a database connection into your VPC. It calls the RDS Data API, which must be enabled on the cluster, and authenticates with an AWS Secrets Manager secret holding the database user's credentials, whose ARN you give the knowledge base. The table needs columns for the ID, the embedding, the chunk text and metadata, with a vector index (for example HNSW). The Data API endpoint plus the secret is the whole access path, so the cluster can stay private.

Dimensions must match. The vector field's dimension is fixed when the index is created and must equal the embedding model's output. Amazon Titan Text Embeddings V2 outputs 1,024 (default), 512 or 256 dimensions; 1,536 belongs to older models such as Titan Embeddings G1 - Text. A mismatch makes every ingestion fail while writing vectors, and the fix is a new index with the right dimension; settings that leave the dimension unchanged cannot fix it.

Parsing and chunking

Parsing and chunking are chosen when a data source is created and cannot be changed afterwards, so a change means a new data source. The default parser extracts text; for PDFs with tables, charts and diagrams, choose a foundation model parser (or Amazon Bedrock Data Automation) so those elements are extracted.

Metadata and sync

Custom attributes for filtering come from a sidecar file in the same S3 prefix named after the source file plus .metadata.json (for example policy.pdf.metadata.json, with a metadataAttributes object). S3 object tags, user-defined object metadata and differently named files are not read. A sync (StartIngestionJob) is incremental: it indexes new and changed files (including changed metadata) and removes deleted ones. Scheduling and automating refreshes belongs to Task 3.3.

Pick the knowledge base API that matches how much of the pipeline you want Amazon Bedrock to run, apply metadata filters on the server for anything security-relevant, and use a structured data store when answers need live aggregations. Skill 3.2.8 tests these choices.

Because RetrieveAndGenerate runs generation itself, you cannot filter its retrieved chunks by your own threshold before the model sees them. Querying the vector index directly also works but rebuilds what Retrieve already manages.

Metadata filters and tenant isolation

A retrieval configuration can include metadata filters (equals, in, greaterThan, andAll, orAll and others) that limit the search to matching chunks before results are returned. For multi-tenant isolation, put an equals filter on the tenant attribute in every call, and set its value on the backend from the authenticated identity (for example the tenant claim in the user's token), never from the question text. Implicit filtering asks a model to derive filters from the user's query, which is useful for relevance but is not a security control, because the user controls the query. Isolation has to be enforced in the retrieval request itself, not requested from the model.

Structured data

Vector retrieval returns similar passages; it cannot sum, group or join, and nightly exports go stale. A knowledge base connected to a structured data store (Amazon Redshift, including data reached through the AWS Glue Data Catalog) converts a natural-language question into SQL, runs it against the live warehouse and returns the result, with no custom text-to-SQL code. Other retrieval strategies, such as reranking and choosing retrieval settings for answer quality, are part of Task 3.1.

New agents are hosted on Amazon Bedrock AgentCore: AgentCore Runtime runs the agent container, AgentCore Gateway publishes tools over the Model Context Protocol (MCP), AgentCore Identity handles credentials, and AgentCore Memory keeps state that outlives a session. Skills 3.2.9 and 3.2.11 test how these parts are configured and where state must live. Amazon Bedrock Agents is now Bedrock Agents Classic and closed to new customers (from 30 July 2026), so it is a choice only for existing agents; Knowledge Bases and Guardrails are unaffected.

AgentCore Runtime

• The agent ships as a container image in Amazon ECR built for linux/arm64 (for example with docker buildx); an x86 image fails to start. The container serves /invocations and /ping on port 8080.
• Each session runs in its own isolated microVM. A session ends after its idle timeout (default 15 minutes) or maximum lifetime (up to 8 hours on microVMs), set by idleRuntimeSessionTimeout and maxLifetime. A session resumed with the same runtimeSessionId after that starts on a new microVM, so in-process variables and local files are gone.
• Network mode: PUBLIC reaches the internet but not your VPC. VPC mode creates network interfaces in your subnets with private IP addresses only, so attaching them to public subnets gives no internet access; outbound internet needs private subnets with a default route to a NAT gateway. An AgentCore interface VPC endpoint carries calls into the AgentCore APIs, not the agent's outbound calls.

AgentCore Gateway and Identity

Gateway turns AWS Lambda functions, OpenAPI or Smithy APIs into MCP tools behind one endpoint that many agents can share, so tools are not copied into each agent. Identity covers two directions. Inbound authorization decides who may invoke the agent (IAM or a JWT from an identity provider such as Amazon Cognito). Outbound authorization lets the agent call other services: OAuth2 credential providers (for example Google) run the user consent flow and keep each user's tokens in a token vault, so the agent gets a fresh access token on later sessions without custom storage or refresh code. Building that yourself means writing token storage, refresh and consent handling.

Where state lives

Choose short-term events when the agent must replay exactly what was said; choose a long-term strategy when it must remember distilled facts such as preferences in new sessions. A Memory resource with no strategy configured produces no long-term records. Deployment pipelines for agents belong to Task 3.3.