AIP-C01 GenAI Cost Optimization: Token Efficiency, Model Selection, Throughput and Caching on Amazon Bedrock

The cost of a generative AI application on AWS is driven by five things: the tokens you send to the model (input), the tokens it generates (output), the price per token of the model you chose, how often you call the model at all, and any capacity you pay for whether or not it is used. Every cost optimization in this topic attacks one of those five, and the skill the exam tests is matching the lever to the driver that a scenario actually shows.

Two traps run through the topic. First, a lever that makes each call cheaper (batch, caching a prefix, a cheaper tier) does not help when the real waste is a call that should not happen at all. Second, anything done after the model responds (trimming text, discarding fields, dropping thinking blocks) saves nothing, because billing happens when tokens are processed and generated.

You track token usage per request from the model response itself, aggregate it with CloudWatch metrics, keep request-level evidence with model invocation logging, and attribute spend to a tenant or workload with tagged application inference profiles. Each source answers a different question, so the right one depends on who needs the number and how fast.

Attribution when everything runs under one role

A multi-tenant application often calls Bedrock from one Lambda function with one role, so neither IAM identity nor a tag on the function can separate tenants. Two mechanisms fill the gap. requestMetadata on a Converse call attaches key-value pairs (up to 16) such as a tenant ID or use case to the request; they appear in the invocation logs, which can then be filtered by tenant. Application inference profiles are resources you create around a model and tag; passing a profile's ARN as the modelId makes that tenant's or workload's usage billable to a tagged resource, and once the tag is activated as a cost allocation tag, Cost Explorer shows the spend separately. Usage metrics answer "how many tokens", not "how many dollars per tenant".

Estimating before you send

Token estimation lets you budget a request before paying for it: the Bedrock CountTokens API (for supported models) returns the input token count a request would use, so an application can trim context or reject an oversized request up front. Remember that tokenizers differ between model families, so a count from one model does not transfer to another.

maxTokens matters for throughput because Amazon Bedrock reserves quota for it up front. When a request starts, Bedrock deducts the input tokens plus maxTokens from your tokens-per-minute (TPM) and tokens-per-day quotas, adjusts as the response is generated, and finally settles the request at its actual usage, returning the unused reservation. A maxTokens value far above what the model ever produces therefore lets far fewer requests start in the same minute, and you see ThrottlingException errors while the tokens actually processed stay well under the quota.

The fix is to right-size maxTokens just above the longest outputs you observe (CloudWatch OutputTokenCount shows the distribution). That costs nothing and increases the concurrent requests that fit in the quota. Retries with backoff smooth bursts but do not change the reservation; Provisioned Throughput or the priority tier add cost to solve a problem that a parameter change removes.

Burndown: quota tokens are not billed tokens

Anthropic Claude models and some other models have an output-token burndown rate greater than 1, typically 5x or more depending on the model version, so check the model card rather than assuming one value. For quota purposes a request settles at:

input tokens + cache write tokens + (output tokens × burndown rate)

Cache read tokens do not count against the quota. Billing still charges the actual tokens; only quota consumption is multiplied. Worked example with different numbers from any practice item: 100 requests a minute, each with 1,000 input tokens and 200 output tokens on a model with 5x burndown, settle at 1,000 + 1,000 = 2,000 quota tokens each, so plan for about 200,000 TPM, although the bill counts only 120,000 tokens. Planning from the billed figure, or from the initial input + maxTokens deduction, gives the wrong answer; so does applying the burndown to maxTokens rather than to actual output.

You control response size by telling the model how long the answer should be, capping it with maxTokens as a safety net, and ending generation at a known marker with stop sequences. Output tokens are usually priced several times higher than input tokens, so unused output is often the single biggest waste.

Stop sequences are the cheapest fix when a template already puts the useful part first and the waste after it: ending at the delimiter removes the trailing text without risking truncation of variable-length useful output. Structured JSON output fixes the format, but it still generates every field you ask for.

Reasoning tokens are output tokens

Extended thinking (on Claude models that support it) generates reasoning before the answer, controlled by budget_tokens (minimum 1,024). Claude 4 models return a summary of the thinking but bill the full thinking tokens as output, so billed output can far exceed the visible response. When an evaluation shows the task is just as accurate without thinking, turning it off removes that cost entirely; a smaller budget only reduces it, and dropping thinking blocks from the stored response changes nothing on the bill.

You cut input tokens by sending the model only what the current request needs: bounded conversation history, compact tool results, fewer but better retrieved chunks, and static prompt content trimmed to what an evaluation shows the task uses. The model is billed for every input token on every call, and in loops and chats the same tokens are resent many times.

Multi-turn chats: sliding window plus summary

A chatbot that resends the full history pays more on every turn, so long sessions cost far more than short ones. A plain sliding window (only the last few turns) bounds the cost but forgets facts given early, such as an account number. The standard pattern is recent turns verbatim plus a running summary that preserves the key facts. A larger context window changes how much fits, not how much you pay; storing history elsewhere and sending only the latest message breaks the conversation.

Agents and tool loops: trim tool results

In a Converse tool-use loop, every tool result stays in the message history and is resent on each later call. Two complementary fixes: make the tool handler return only the fields the agent uses (cut at the source), and replace older tool results with short summaries once they have been used (compact what remains). Prompt caching can make a resent prefix cheaper to read, but the tokens are still sent; it does not reduce the input token count.

RAG: fewer, better chunks

Each retrieved chunk becomes input. In Amazon Bedrock Knowledge Bases, numberOfResults in the vector search configuration sets how many chunks are retrieved and passed to generation. If an evaluation shows answers need only the top few, lowering numberOfResults is a one-value change. If the relevant chunk often ranks low in vector search, lowering it hurts recall; instead keep a wide retrieval and add a reranker model with numberOfRerankedResults set to a small number, so only the most relevant chunks reach the model. A reranker that passes on as many results as were retrieved prunes nothing. Embedding dimensions affect the vector store, not the generation prompt.

Prompt compression: trim static content

Prompt compression removes tokens that carry no information for the task: unused fields in a data record, nulls and pretty-printing whitespace in JSON, redundant few-shot examples, and whole reference documents when each request needs one section of them (retrieve the relevant part instead). Base the cut on an offline evaluation so accuracy is preserved; removing all examples or instructions without testing is not compression, it is a quality gamble. Byte-level compression (gzip, Base64) does not help: a text model cannot read it, and encoded text usually costs more tokens. Moving text from the system prompt to a user message changes nothing about its token count.

You choose a cost-effective model by measuring the cost per successful task on your own representative data, for every candidate that meets the quality bar, not by comparing list prices per token. A cheaper model that fails more often can be the more expensive system once downstream costs (human review, retries, escalations) are included.

Measuring price to performance

• Quality first: run an evaluation (for example an Amazon Bedrock model evaluation job, with your own rubric or an LLM-as-a-judge) on a representative sample to see which models meet the requirement. Public leaderboards measure general tasks, not yours.
• Cost from each model's own tokens: tokenizers differ between model families and some models are more verbose, so compute each model's cost from the tokens it actually used on the same sample. Applying a new model's prices to the current model's token counts is wrong.
• Total cost per outcome: add the cost of handling errors. Example with made-up figures: model A costs $0.010 per item and is right 99% of the time; model B costs $0.006 and is right 90%; each error costs $0.10 to fix by hand. A = 0.010 + 0.01 × 0.10 = $0.011; B = 0.006 + 0.10 × 0.10 = $0.016. The cheaper call is the costlier system.

Tiered model usage

Most workloads mix easy and hard requests. Tiered usage sends simple requests to a small, cheap model and only complex ones to a large model, using Amazon Bedrock Intelligent Prompt Routing, a classifier, or a cascade that escalates when the small model's answer fails a check. Routing works when the small model can handle a large share of traffic on its own; if it is weak on the task, most requests escalate and the savings vanish. (Routing and deployment patterns are covered in depth in the model deployment lesson.)

Model Distillation: make the small model good at your task

Amazon Bedrock Model Distillation transfers a narrow task from a large teacher model to a smaller, cheaper student in the same family. You supply prompts, and the teacher generates the responses used to fine-tune the student, so no human labeling is needed; or you point it at your existing model invocation logs, filtered by requestMetadata (for example a use-case key), to reuse prompt-response pairs the teacher already produced in production. The result is a custom model that you serve through Provisioned Throughput or on-demand custom model deployment where supported. It fits stable, high-volume tasks where the student falls short with plain prompting and few-shot prompting makes it too expensive. Only specific teacher and student pairs are supported, so check the model list. Compare with fine-tuning (needs labeled examples you create), continued pre-training (teaches domain text, not the teacher's answers) and prompt routing (chooses between models, does not improve either).

Service tiers are chosen per request with the service_tier parameter (standard is the default; the on-demand quota is shared by standard, flex and priority, while a Reserved tier reservation is separate capacity). Use batch inference for bulk work nobody waits on, the flex tier for latency-tolerant requests you still send one at a time, standard on-demand for interactive traffic with variable load, and Provisioned Throughput only when monitoring shows steady, high utilization or when a custom model requires it. The deciding facts are whether a user is waiting, how steady the load is, and how much operational work each option needs.

Batch inference in practice

A batch job takes JSONL records (one model input per line) from Amazon S3 and writes outputs back to S3. You submit it with CreateModelInvocationJob, one job per model, and react to completion with Amazon EventBridge job state change events, so there are no servers, queues or throttling logic to run. That makes it both the cheaper and the lower-overhead choice for scheduled bulk work with a deadline measured in hours or days.

Utilization and capacity planning

Capacity planning means sizing from measured peak traffic (requests per minute × quota tokens per request, including burndown) and watching utilization over time in CloudWatch. Provisioned Throughput pays off only when it stays busy. If traffic is bursty with long idle periods and the on-demand quota covers the peak, letting the commitment lapse and moving to on-demand removes the idle cost without operational work; recreating no-commitment throughput every day adds operations and pays higher hourly rates. Conversely, when you already pay for Provisioned Throughput that sits idle at night, pacing deferrable work into those idle hours uses capacity that is already paid for. Batch jobs cannot target provisioned models, and switching to the base model loses a fine-tuned model's behaviour.

Auto scaling for self-hosted models

Models you host on Amazon SageMaker AI endpoints are sized in instances, not tokens. Configure Application Auto Scaling with a target-tracking policy (for example on invocations per instance) so capacity follows demand, and use scale-in to zero for inference components where traffic has idle periods. Bedrock on-demand inference needs no scaling configuration; its limits are quotas.

Prompt caching stores a repeated prefix of a request (tools, system prompt, documents, examples) so later requests that start with exactly the same prefix read it from cache at a reduced price and with lower time to first token. It still calls the model every time; it makes a repeated part of the input cheaper and faster, nothing more.

Bedrock offers two kinds: implicit caching, which some models (for example Amazon Nova and OpenAI models) apply automatically on a best-effort basis, and explicit caching, where you mark checkpoints. The exam scenarios are about explicit caching.

Mechanics

• Checkpoints: in the Converse API you add a cachePoint block after the content to cache; with InvokeModel on Claude you use cache_control. Everything before the checkpoint forms the cached prefix. Models limit how many checkpoints a request may have (four for Claude). Automatic caching is the alternative to placing checkpoints yourself: a single top-level cache_control field (in the InvokeModel or Messages body, or in additionalModelRequestFields with Converse) makes Claude place the checkpoint on the last cacheable block and move it forward as the conversation grows.
• Prefix order: the prefix is built in the order tools → system → messages, and it must be byte-identical across requests. A changing value at the start (a timestamp, a customer name) makes every request a new prefix, so you see cache writes on every call and zero reads. Put static content first and dynamic content after the last checkpoint.
• Minimum size: each model has a minimum number of tokens per checkpoint (minimums vary by model, from 512 to 4,096 tokens; 1,024 for several Claude models, so never assume one value), counted cumulatively over everything before the checkpoint. Below it, the request succeeds but nothing is cached, and both cacheWriteInputTokens and cacheReadInputTokens stay 0. Moving the checkpoint after more static content (such as fixed few-shot examples) fixes it.
• TTL: for Claude models the default lifetime is 5 minutes (some other models on Bedrock use a different TTL, for example 30 minutes) and it resets on every cache hit. Supported Claude models also offer a 1-hour TTL, set on the checkpoint. Automatic caching keeps the default TTL unless you set the longer one.
• Not everywhere: batch inference does not use prompt caching, and cross-Region inference can route requests to Regions that do not hold the cached prefix, which may increase writes.

Economics: writes cost, reads save

Writing to the cache is charged at a cache-write rate that can be higher than the normal input rate; reading is charged at a much lower rate. Caching therefore pays off only when a prefix is read several times within its TTL.

Read cacheReadInputTokens and cacheWriteInputTokens in the response usage to diagnose: all writes and no reads means a changing or expired prefix; zeros for both means the prefix is below the minimum.

Response caches avoid unnecessary FM invocations by returning a stored answer when a request has effectively been seen before. Unlike prompt caching, a cache hit skips the model entirely. The four techniques differ in how they decide that two requests are "the same".

Deterministic request hashing

Key the cache (for example in DynamoDB or ElastiCache) on a hash of everything that affects the output: model ID, inference parameters such as maxTokens and temperature, the normalized prompt, and a version of the underlying data (such as a snapshot date) so keys change when data does. Normalize before hashing: serialize with sorted keys and consistent formatting, and exclude per-call fields that never reach the model, such as trace IDs. Hashing only the prompt text merges requests that differ in model or parameters; a longer TTL never makes unique keys match.

Semantic caching

Embed each incoming question, search stored question embeddings in a vector-capable store (for example Amazon ElastiCache or Amazon MemoryDB with vector search, or Amazon OpenSearch Service), and return the stored answer when similarity exceeds a threshold. This catches paraphrases that exact hashing misses; normalizing case and punctuation does not. Two rules keep it correct: scope entries by any context that changes the answer (plan, Region, language, user role) and filter the vector search on it, because the question text alone may not determine the answer; and keep entries fresh with a TTL plus invalidation when the source content changes. Lowering the threshold raises hit rate at the cost of more wrong matches.

Result fingerprinting

Store a content hash (for example SHA-256 of the document text) with each generated result, and skip regeneration when the hash is unchanged. Timestamps and last-modified times are unreliable when tools re-upload unchanged files. The fingerprint must cover every input to the result: combine the content hash with the prompt template version and the model ID so a template or model change regenerates exactly the affected results automatically. An S3 ETag is still content-only.

Edge caching

When every visitor sees the same generated text, generate it once and serve it from Amazon CloudFront. CloudFront caches responses to GET and HEAD requests, so expose the content through a GET with the identifying value in the path or a query string, not a POST body. The cache policy defines the cache key: include only the values that change the content, and leave out session cookies and marketing query strings such as utm_source, which otherwise split one page into thousands of cache entries. An origin request policy controls what is forwarded to the origin but does not change the cache key. API Gateway stage caching is Regional and by default caches GET methods only, so it does not replace edge caching for a global audience.