Optimising ML and GenAI Inference Cost and Performance on AWS: Instances, Savings Plans, Tokens and Vectors (MLA-C02)

Choose the instance family from the resource the model actually runs out of, not from the model's name or the instance you trained on. A SageMaker AI real-time endpoint pays for every vCPU, GiB of RAM and accelerator on the instance every hour, so any resource that sits idle is pure cost. Read the endpoint's utilisation metrics first, then move to the family that supplies the bottleneck resource and little else.

Two rules catch most mistakes. First, an accelerator the model does not use (a classical model on ml.g5 with GPUUtilization near zero) is always the first thing to remove. Second, adding a bigger instance of a family that already holds the bottleneck is usually not cheaper than moving to the right family: an ml.m5.4xlarge costs the same as two ml.m5.2xlarge, so consolidating changes nothing.

GPU memory decides LLM fit

For self-hosted LLMs the binding limit is GPU memory. Weights in FP16 or BF16 take about 2 bytes per parameter, so a 13-billion-parameter model needs about 26 GB for weights alone, plus headroom for the KV cache that grows with batch size and context length. Useful figures: NVIDIA T4 (g4dn) has 16 GB, A10G (g5) and L4 (g6) have 24 GB, and L40S (g6e) has 48 GB per GPU. The single-GPU sizes of a family all share the same GPU: moving from ml.g5.xlarge to ml.g5.8xlarge adds vCPUs and host RAM, not GPU memory. The larger multi-GPU sizes (such as ml.g5.12xlarge with four GPUs) fit bigger models only by splitting them across GPUs with tensor parallelism. When a model just overflows a 24 GB GPU, the smallest instance with a 48 GB GPU is usually cheaper than a multi-GPU size. Quantisation (8-bit or 4-bit weights) is the other lever, at some cost in quality.

The cost comparison between Inferentia, Graviton and GPU instances for a given deployment, and model compilation, are covered in the Task 3.1 lesson on deployment; this lesson focuses on rightsizing a running workload.

Confirm the choice with SageMaker Inference Recommender

Amazon SageMaker Inference Recommender load-tests a model on candidate instance types and reports latency, throughput and cost per inference for each, so you do not have to build a benchmarking harness. It works on a model package version in SageMaker Model Registry (or a model you describe with its container, artefact and sample payload).

Use a default job first, then an advanced job when the requirement specifies the traffic pattern to simulate or a latency target the test must enforce: only an advanced job takes traffic phases and stopping conditions. Benchmarking belongs before launch, not on production traffic. Inference Recommender is a pre-deployment tool: SageMaker Model Monitor watches drift after deployment, SageMaker Debugger profiles training jobs, and AWS Cost Explorer rightsizing recommendations cover EC2 instances, not SageMaker AI endpoint sizing.

Start with the endpoint's own CloudWatch metrics to decide whether the time is spent inside SageMaker AI or somewhere else, then trace the rest of the request path with AWS X-Ray.

Read the metrics as a budget. If users wait seconds but ModelLatency plus OverheadLatency is only tens of milliseconds, the missing time is outside the endpoint: an API Gateway integration, a Lambda cold start or another downstream call. Scaling or profiling the endpoint will not help. Turn on X-Ray active tracing on the API Gateway stage and the Lambda function; each trace then shows a segment per hop with its duration. Also note that MemoryUtilization is host RAM: before moving a model to a GPU with less memory, check GPUMemoryUtilization, because low GPUUtilization says nothing about whether the weights fit.

On a serverless endpoint, slow first requests after a quiet period that coincide with ModelSetupTime spikes (while ModelLatency is steady) are cold starts. Provisioned concurrency keeps environments initialised and is billed while configured; raising memory size or maximum concurrency does not remove setup time.

X-Ray sampling is decided at the entry point

X-Ray does not trace every request: sampling rules (the default traces the first request each second plus 5% of the rest) decide which requests are recorded, and the decision is made by the first instrumented service a request reaches and passed downstream in the trace header. A rule that matches only a downstream service therefore never applies when that service is always called by an instrumented upstream. To trace more of one path, create a higher-priority rule that matches the entry service and the specific URL path, rather than raising the default rule (which samples everything more) or tuning a rule on the downstream service.

Troubleshooting agents with AgentCore Observability

Amazon Bedrock AgentCore Observability shows, for each agent session, a trace whose spans record every model invocation, tool call and step with its duration and order, alongside runtime metrics, in the CloudWatch generative AI observability console. That per-session timeline is what you need when some sessions are slow: aggregate metrics blend all sessions together, and a log of model calls alone shows only one kind of step, so neither reconstructs the sequence of steps that made one session slow.

Agents hosted on AgentCore Runtime publish metrics automatically. Spans need two things:

• Instrumentation in the agent: the AWS Distro for OpenTelemetry (ADOT) SDK in its dependencies, with the agent started through opentelemetry-instrument (frameworks such as Strands emit spans through it). The separate ADOT Collector is not supported for AgentCore agents.
• CloudWatch Transaction Search enabled once per account, so trace segments are ingested into CloudWatch Logs as spans. If metrics appear but no traces appear for any session, this is the missing step; sampling rules change how many requests are traced, not whether any are stored.

The monitoring of answer quality, online evaluations and the GenAI observability prebuilt views as quality tools belong to Task 4.1; here the focus is finding where time and resources go.

A useful performance dashboard plots the right statistic, covers every account and resource automatically, and does not copy data. Three CloudWatch features do most of the work.

• Percentile statistics. Latency complaints are about the slowest requests. Average over thousands of fast requests stays flat when a few take seconds; plot p90, p95 or p99 to show the tail. Sum and SampleCount measure volume, and a longer period smooths spikes away.
• Cross-account observability. Link workload (source) accounts to a central monitoring account with CloudWatch cross-account observability (Organizations or individual links). Dashboards, alarms and queries in the monitoring account can then use the linked accounts' AWS/SageMaker, AWS/Bedrock and other metrics, logs and traces directly, with no metric streams, log subscriptions or extra services. Per-account dashboards with links between them are not a single view.
• Metric math and SEARCH. A widget built from fixed metric IDs only shows the resources that existed when it was made. A SEARCH expression, for example SEARCH('{AWS/SageMaker,EndpointName,VariantName} MetricName="ModelLatency"', 'Average', 60), re-runs when the dashboard loads, so new endpoints that match appear without anyone editing it. Other metric math (such as SUM, ratios or METRICS()) derives values like error rate or tokens per invocation.

Useful generative AI metrics in AWS/Bedrock include Invocations, InvocationLatency, InvocationThrottles, InputTokenCount, OutputTokenCount, CacheReadInputTokenCount and CacheWriteInputTokenCount, by ModelId. For alerting without fixed thresholds, a CloudWatch alarm based on anomaly detection learns a metric's hourly and daily pattern and fires within minutes when, for example, OutputTokenCount leaves its expected band.

Capacity optimisation means running just enough instances, in the right shape, at the right time, while meeting the availability requirement. These levers recur:

The trade-off is always explicit: a multi-model endpoint accepts a load delay when an evicted model is called; scale-to-zero accepts a cold start; a scheduled floor accepts paying for instances slightly before they are busy. Choose the lever whose penalty the scenario says is acceptable. Target-tracking metrics and scaling policy design are taught in the Task 3.2 lesson on endpoint scaling.

AWS cost tools split into attribution (who spent it), forecasting and limits (will we overspend, and what happens then) and anomaly alerts (is something unusual). Match the tool to the job.

Two attribution details matter for Bedrock. Bedrock also supports cost allocation by IAM principal tags, which separates teams when each team calls through its own role or user; when teams all call through one shared role, that role's tags cannot tell them apart, and per-team application inference profiles are the dependable split. And tags only become cost dimensions once activated: tagging resources alone never makes a tag appear in Cost Explorer. Budgets actions cannot stop SageMaker AI resources directly, so a deny policy on the create actions is the way to freeze new spend while leaving running work alone.

Commitment discounts pay off for steady usage, and the right one depends on which service the usage runs on. For SageMaker AI the commitment product is the SageMaker Savings Plan.

Size the commitment to the baseline. A Savings Plan commitment is applied hour by hour, and any commitment not used in an hour is lost. If usage never drops below a steady baseline but rises for a few hours a day, commit to cover the baseline and run the peak at on-demand rates. Committing to the daily average or the peak looks like more discount, but the unused commitment in every off-peak hour costs more than the extra discount earns. A commitment only discounts capacity; it never fixes capacity that should not exist, such as hundreds of idle endpoints or an oversized GPU instance.

The cost of FM inference depends on how you buy capacity: per token, per tier, per batch, per reserved unit or per instance-hour. Each option has a different break-even point.

The service tier is set per request, and not every model supports every tier. Two scenario patterns follow. When jobs must remain ordinary synchronous calls without reworking input into S3 files, Flex is the discount that fits; batch inference is cheaper only if the work can move to S3 jobs. When a self-hosted model sits on a GPU instance at a few percent utilisation, most of the hourly charge is waste; if the same model is offered as a Bedrock on-demand model in the required Region, per-token billing removes the idle cost, whereas a Savings Plan would only discount the waste and SageMaker serverless inference is not an option for GPU models.

Cross-Region inference changes where requests run, not just capacity. A geographic inference profile routes requests to other Regions in the same geography (priced as the source Region), and global profiles can route anywhere; neither satisfies a rule that requests must stay in one Region. Model selection, prompt routing and the on-demand versus Provisioned Throughput deployment decision are covered in the Task 2.1 and 3.1 lessons.

An agent costs two separate things: model tokens (billed by Bedrock per model call) and runtime compute (billed by AgentCore Runtime for the session's vCPU and memory). Monitor them separately, because a looping agent inflates tokens while an idle session inflates memory.

AgentCore Runtime publishes CloudWatch metrics including Invocations, Session Count, Latency, CPUUsed-vCPUHours and MemoryUsed-GBHours, with dimensions for the service, the agent resource and the endpoint name, plus per-session usage logs. To answer which agent endpoints drive the Runtime bill, group the vCPU-hour and GB-hour metrics by endpoint; invocation counts, session counts or latency multiplied by calls are proxies that ignore how Runtime bills.

How Runtime bills: CPU is charged only for active processing (time spent waiting on a model response or other I/O is not CPU time), while memory is charged for as long as the session exists, including idle periods, until it terminates. A session ends when the client calls StopRuntimeSession, when it has been idle for idleRuntimeSessionTimeout (default 900 seconds) or when it reaches maxLifetime (default 28,800 seconds).

• Sessions that last far longer than conversations, with low vCPU use, mean memory is the cost: call StopRuntimeSession when the user ends the chat, and lower idleRuntimeSessionTimeout for clients that disappear without calling it.
• The session lifetime settings and explicit stops control memory cost; token limits control model cost, not session memory.
• Reuse one runtimeSessionId for every turn of a conversation: each new ID starts a new session, which loses the conversation's context and multiplies session start-ups.

For the token side, watch OutputTokenCount and Invocations per model, cap agent iterations in the framework, and alert on anomalies, as described above.

Token cost and throttling are both driven by how many tokens each request sends, asks for and repeats. Four levers cover most cases.

• Right-size the maximum output tokens. When a request starts, Bedrock deducts the input tokens plus the requested maximum output tokens from the tokens-per-minute quota, and only corrects to the actual output when the response finishes. A service whose maximum is set far above what it typically writes reserves many times what it uses and is throttled while CloudWatch shows usage far below the quota. Because the reservation is set by the requested maximum, setting that maximum just above the longest expected response is what fixes it without a quota increase. For some models, such as recent Anthropic Claude models, each output token also counts against the quota at a higher burndown rate (5x) than an input token, which makes oversized outputs burn quota even faster.
• Prompt caching for repeated prefixes. Put a cache checkpoint (cachePoint in the Converse API) after the content that is identical on every request, such as system instructions and tool definitions, and before the part that changes. Cache reads are billed at a reduced rate. A checkpoint placed after the per-request message produces a cache write on every request and no reads (CacheWriteInputTokenCount rising, CacheReadInputTokenCount at zero). Each model has a minimum number of tokens per checkpoint (for example 1,024 for Claude Sonnet 4.5); a shorter prefix is silently not cached and the cache counters stay at zero. Caching works with Converse, ConverseStream and InvokeModel on on-demand inference, and cache entries expire after a short idle period.
• Trim the conversation history. Resending the full history makes InputTokenCount grow with every turn. Keep recent turns verbatim and replace older ones with a running summary.
• Retrieve fewer, better passages. Each retrieved passage adds input tokens on every turn. Lower numberOfResults on the knowledge base Retrieve call after testing that answers stay good.

Input tokens fall only when less text is sent, so history length and retrieval size are the input-side levers. Choosing a cheaper model or a prompt router for simple requests is the other big lever (Task 2.1).

RAG systems add two AI-specific cost lines: computing embeddings (tokens through an embedding model) and storing and searching vectors. Both scale with how much text you embed and how big each vector is.

Embedding computation

• Sync incrementally. A Bedrock knowledge base ingestion job on an existing data source detects added, modified and deleted documents and embeds only those. Deleting and re-creating the data source forces every document to be parsed, chunked and embedded again.
• Use batch inference for bulk one-off embedding. Amazon Titan Text Embeddings V2 supports Bedrock batch inference: JSONL records in S3, embeddings written back to S3, at the discounted batch rate. The per-token price is set by how Bedrock is invoked, not by where the calling code runs.
• Embed less text. Chunk overlap repeats tokens: the higher the overlap, the more text is embedded more than once, so keep it modest. S3 inclusion prefixes keep superseded or irrelevant objects out of ingestion. Semantic chunking calls a foundation model to find boundaries, which adds cost; hierarchical chunking changes what retrieval returns rather than how much is embedded. Requesting fewer output dimensions does not lower the input-token price.

Vector storage

OpenSearch Serverless capacity is set per collection group: collections in one group share its OCUs, each group has minimum and maximum OCUs for indexing and search, the minimum can be 0 (accepting slower first requests after idle periods), and a group holds one collection type. Pooling idle vector collections into one group and setting minimums to 0 cuts the bill without capping daytime capacity; lowering the maximum would cap it.

To shrink the index itself, shrink each vector. Titan Text Embeddings V2 outputs 1,024, 512 or 256 dimensions, and can return binary embeddings (1 bit per dimension instead of a 32-bit float) that OpenSearch Serverless stores in a binary vector index. Both need re-embedding and a matching index, and both trade a little retrieval precision.