AIP-C01 GenAI Performance: Latency, Retrieval Tuning, Token Throughput and Inference Parameters

An LLM call spends its time in two phases: prefill, where the model reads the whole prompt before it can produce anything, and decode, where it generates output one token at a time. Prefill sets the time to first token (TTFT) and grows mainly with input length; decode sets how long the rest of the answer takes and grows with output length. For a streamed call the arithmetic is:

InvocationLatency ≈ TimeToFirstToken + OutputTokenCount ÷ output tokens per second (OTPS)

Profiling prompt-completion patterns means reading these numbers together instead of looking at total latency alone. Amazon Bedrock publishes runtime metrics to the AWS/Bedrock CloudWatch namespace by model ID, with no invocation logging required: Invocations, InvocationLatency, InputTokenCount, OutputTokenCount, throttles and errors, and TimeToFirstToken. TimeToFirstToken exists only for streaming operations (ConverseStream and InvokeModelWithResponseStream); an application that calls Converse or InvokeModel produces InvocationLatency, which runs to the last token, and leaves a TTFT graph empty.

Reading the profile

A worked check: suppose p50 latency rose from 6 to 11 seconds while TTFT held at about 1 second and the average answer grew from 300 to 600 tokens. OTPS is 300 ÷ 5 = 60 before and 600 ÷ 10 = 60 after. The model is no slower; the answers are twice as long. Settings that act on output, such as maxTokens and temperature, do nothing for a prefill problem, and settings that act on input do nothing for a decode problem, so diagnose first.

You make a GenAI application responsive by removing model calls from the user's path where you can, and making the remaining calls start sooner, run faster or run side by side. Skill 4.2.1 names five techniques, and each solves a different part of the latency-cost trade-off.

Pre-computation for predictable queries

If you can predict what will be asked and tolerate some staleness, generate the answers once and serve them by lookup. Product-review summaries that may be a day old, or morning summaries for a known list of tickers whose source data is complete before the peak, are pre-computation jobs: a scheduled workflow (or a Bedrock batch inference job for large sets) writes each answer to a key-value store, and a router sends matching requests to the store and everything else to the model. This beats the alternatives because they all still make a live call. A cache filled on first view still makes the first visitor wait, and with a long tail of rarely viewed items most views are first views. A semantic cache only helps after someone has asked, and questions that differ only by an entity name (one ticker versus another) can embed close together and return the wrong cached answer. Reserved capacity removes throttling but not generation time. (Prompt caching, semantic caching and their cost effects are covered with Task 4.1.)

Latency-optimized inference

Amazon Bedrock latency-optimized inference serves a supported model on faster infrastructure when the request sets "performanceConfig": {"latency": "optimized"} on Converse, ConverseStream, InvokeModel or InvokeModelWithResponseStream. Requests default to standard. It needs no model change, no retraining and no capacity commitment, but it is limited to a small set of models (for example Amazon Nova Pro and Anthropic Claude 3.5 Haiku), reached through cross-Region inference profiles from specific Regions, and AWS still labels it a preview. It has its own usage quota: once that quota is reached, Bedrock serves further requests at standard latency and charges standard rates, without an error. The latency mode that actually served a request is reported in the response and in CloudTrail, so a peak hour with some slow replies, no failures and some usage billed at standard rates points to that fallback, not to throttling. Throttling surfaces as ThrottlingException errors that the SDK retries with the same settings.

Keep its job distinct from nearby features. Latency-optimized inference changes how fast a supported model generates each response. Capacity features (Provisioned Throughput, cross-Region inference profiles) change how much traffic can be served and where it runs, which matters for throttling and availability rather than per-request speed. Compute-side features such as Lambda provisioned concurrency change how fast your own code starts, so they help only when traces show the delay is in your code rather than in model processing.

Parallel requests

Independent FM calls made one after another add up; run them in parallel and you wait only for the slowest. In AWS Step Functions put independent Bedrock tasks in the branches of a Parallel state, or in a Map state with enough concurrency, then merge the results; in code, issue concurrent SDK calls. Four independent 5-second prompts take about 20 seconds in sequence and about 5 in parallel. Two principles follow. Concurrency is what saves the time, so any setting that limits in-flight work to one (a Map whose MaxConcurrency is 1, for example) keeps the calls sequential. And a single model request is itself sequential: merging independent prompts into one request still generates every part token by token in one stream, and a crowded prompt tends to lower quality. The workflow type (Standard or Express) governs duration limits and execution semantics, not how fast each call runs. Steps that need an earlier result, such as a final synthesis over partial answers, must still wait for it.

Response streaming

Streaming does not make the model finish sooner; it lets the user see or hear the answer while the rest is generated. ConverseStream (or InvokeModelWithResponseStream) returns tokens incrementally, so a chat UI renders the first words within about the TTFT, and a voice assistant can hand each complete sentence to speech synthesis instead of waiting for the full reply. Streaming and latency-optimized inference combine well: one makes tokens arrive sooner, the other lets the user start consuming them earlier. The best design often stacks techniques: parallelize what is independent, stream what the user reads. (Delivering the stream to browsers over WebSockets or server-sent events is Task 2.4.)

A useful FM benchmark replays representative prompts at production concurrency, from the target Region, using the same API mode as production, and compares percentiles of the metrics users actually feel. Latency depends heavily on prompt length (prefill), on how many requests are in flight and on the network path, so ten short prompts run one at a time from a laptop say almost nothing about a 50-user, long-prompt workload, however many times they are repeated.

• Workload: real or recorded production prompts, so input and output lengths are realistic.
• Load: the expected peak concurrency, not one request in flight; queuing and tail latency only appear under load.
• Mode: streaming calls if the application streams, because only streaming calls expose TTFT; a non-streaming test of the same load measures total response time instead.
• Metrics: for a streaming chat, TTFT (how quickly the answer starts) and output tokens per second (whether a long answer keeps pace with reading), each at a high percentile such as p90 or p99. An average of end-to-end latency mixes start-up with generation and hides slow tails.
• Not speed metrics: token counts, published context windows and maximum output limits are capability and cost facts. A Bedrock model evaluation job (including LLM-as-a-judge) measures answer quality, not speed; use it alongside, not instead of, a latency benchmark.

Query preprocessing improves retrieval by fixing what the query says before it reaches the index: turning stated facts into filters, splitting compound questions, rewriting context-dependent follow-ups and normalizing identifiers. It is usually cheaper and more effective than tuning the index, because no index setting can recover information the query does not carry.

Amazon Bedrock Knowledge Bases supports several of these without custom code. Metadata filters (equals, in, andAll, orAll and others) go in the retrieval configuration. Query decomposition is switched on for RetrieveAndGenerate by setting queryTransformationConfiguration type QUERY_DECOMPOSITION in the orchestrationConfiguration; the knowledge base then generates and retrieves for each sub-query. overrideSearchType can be set to HYBRID or SEMANTIC for supported vector stores, and a reranking model can reorder results (which improves order, not which documents are searched). After retrieval, sending only the few most relevant chunks keeps prompts short, which directly lowers TTFT.

In Amazon OpenSearch Service you implement weighted hybrid search with a hybrid query that holds a lexical sub-query and a vector sub-query, plus a search pipeline with a normalization-processor that normalizes each sub-query's scores and then combines them with weights. The pipeline is what makes the scores comparable: a hybrid query relies on it to normalize and combine, so without one there is no normalization step and nothing to weight.

Why a bool query is not enough

Putting a match query and a knn query in one bool should clause adds their raw scores. BM25 scores have no fixed upper bound and change with each query's terms, while vector similarity scores sit in a narrow range, so the lexical side usually dominates, and a fixed boost on the vector clause over-corrects for some queries and under-corrects for others. A rescore stage only reorders documents the first query already found, so semantic matches without the keywords never appear.

The normalization processor

• Normalization technique — min_max (rescales each sub-query's scores to 0–1) or l2 (divides by the norm of that sub-query's scores); newer versions also offer z_score, which can be combined only with arithmetic_mean (the other two techniques work with all three combinations). Each sub-query is normalized independently, and both min_max and l2 are scale-invariant, so multiplying a sub-query's scores by a constant boost has no effect after normalization.
• Combination technique — arithmetic_mean, geometric_mean or harmonic_mean.
• Weights — combination.parameters.weights, a list whose positions match the order of the sub-queries in the hybrid query and which sums to 1.0. With the keyword query first and the k-NN query second, [0.4, 0.6] gives the semantic side 60% of the combined score; listing the sub-queries the other way round flips which side each weight applies to.

"normalization-processor": {
  "normalization": { "technique": "min_max" },
  "combination": {
    "technique": "arithmetic_mean",
    "parameters": { "weights": [0.4, 0.6] }
  }
}

Tune weights against a relevance test set, not by eye. In Bedrock Knowledge Bases backed by a supported vector store, the managed equivalent is the HYBRID search type, which handles the combination for you.

You tune a vector store by applying filters during the approximate search rather than after it, and by trading surplus recall for latency with search-time parameters before touching build-time ones. Both appear in Skill 4.2.2 (index optimization) and Skill 4.2.6 (vector database query optimization).

Filtering: efficient filtering versus post-filtering

If the filter sits in a bool query around the knn clause, OpenSearch finds the global top k neighbours first and then removes the ones that fail the filter. With a restrictive filter, such as one tenant in a multi-tenant index, few or none survive. Efficient k-NN filtering puts the filter in the filter parameter inside the knn clause; supported for the Lucene engine and for Faiss (HNSW, and IVF on newer versions), it applies the filter during the search and returns k results whenever at least k matching documents exist, while keeping approximate-search latency. Raising k only makes post-filtering less likely to come up short, at a latency cost; an exact script_score search over the filtered set is accurate but its cost grows with the number of matching documents.

HNSW parameters: which need a rebuild

The general rule: search-time settings are the cheap dial, build-time settings need a reindex, and each moves recall and latency in the same direction. When measured recall is comfortably above target and p99 latency is over budget, lower ef_search in steps against a fixed query set, measuring both, until recall nears the target. Exact (brute-force) search sits at the far end of the same trade-off, with perfect recall at the highest cost. And ef_search only governs how thoroughly the graph is explored; it cannot recover results that a filter removes after the search.

Bedrock on-demand throughput is limited by requests per minute and tokens per minute (TPM) quotas, and the TPM quota is charged in two steps. At the start of each request Bedrock deducts input tokens + max_tokens and throttles if that would exceed the quota. When the request completes it adjusts the deduction to InputTokenCount + CacheWriteInputTokenCount + OutputTokenCount × burndown rate and returns the rest. Cache reads do not count toward the quota. The burndown rate is model-specific: most models count output tokens 1:1, while Anthropic Claude and some other models burn output tokens at a multiple (5× for older Claude models, 10× or 15× for the newest), so check the model's current rate in the Bedrock documentation before planning. Billing is separate: you pay for the tokens actually processed, not for the burndown.

Worked example, on a model with 5× burndown: 2,500 input tokens, max_tokens 6,000, a 300-token answer. Deducted at start: 2,500 + 6,000 = 8,500. Consumed at completion: 2,500 + 300 × 5 = 4,000. Billed: 2,800 tokens (input plus actual output).

Token processing optimization: size max_tokens to the real output

Because the reservation uses max_tokens, an oversized value throttles concurrent requests long before actual consumption reaches the quota. Suppose requests average 3,000 input tokens and set max_tokens to 16,000, but answers never exceed about 900 tokens: each request reserves about 19,000 tokens while consuming a fraction of that, so token-count graphs sit well under the quota while ThrottlingException errors pile up. The CloudWatch EstimatedTPMQuotaUsage metric does not settle this either: AWS describes it as an estimate that does not reflect the start-of-request reservation, and advises against using it alone for quota or capacity planning. Setting max_tokens slightly above the real p99 output length brings reservations close to real use at no cost. In general, fix the reservation before buying headroom: quota increases and reserved capacity add approval steps or cost without removing the waste, and input-side savings such as prompt caching shrink only the smaller part of an oversized reservation.

Capacity planning in tokens

Plan TPM as peak requests per minute × (input tokens + cache writes + output tokens × burndown), then add headroom, and check that the start-of-request reservation (input + max_tokens) is not the binding figure for your concurrency. At 300 requests per minute with 3,000 input and 500 output tokens on a 5× model: 3,000 + 500 × 5 = 5,500 tokens per request, about 1.65 million TPM. Counting output 1:1 understates the need (1.05 million here); applying the burndown to max_tokens instead of real output overstates it. Once live, keep the plan honest by watching InputTokenCount and OutputTokenCount per model: utilization monitoring of prompt and completion patterns shows when prompts or answers grow and the plan needs revisiting. (Choosing and sizing Provisioned Throughput for cost is Task 4.1; alarms and dashboards are Task 4.3.)

You keep high-volume FM work within quota by moving offline volume to batch inference and by bounding how many calls run at once, then retrying the few throttles that remain with exponential backoff and jitter.

Batch inference versus on-demand

Amazon Bedrock batch inference reads prompts as JSONL records from Amazon S3, processes them asynchronously under its own job quotas and writes results back to S3, at a lower price than on-demand for supported models. It suits large jobs with hours of slack, such as re-describing a catalog overnight or re-tagging an archive within 24 hours, and it keeps that volume off the on-demand quota that interactive users share. Batch jobs queue and have no guaranteed per-job completion time, and a job needs a minimum number of records, so work with a minute-level deadline (an article that must be tagged before publishing in 15 minutes) stays on on-demand calls. Making an offline job's on-demand calls faster, streamed or more numerous does not remove the contention; more workers concentrate the same load and throttle users harder.

Concurrent invocation management

Unbounded fan-out is the usual cause of throttling bursts. Cap concurrency where the work enters, at roughly the sustainable request rate × average call duration:

• Step Functions Map state (inline or distributed): set MaxConcurrency to a level the quota sustains and add a Retry on throttling errors with exponential backoff. Only those two settings change the outcome: Bedrock rejects a throttled call immediately rather than queuing it, so error handling that merely records the failure, or timeouts and workflow types that leave the number of in-flight calls unchanged, do not reduce throttling.
• Lambda with an SQS event source mapping: set the mapping's maximum concurrency, which limits how many concurrent invocations the queue drives, so excess messages simply wait in the queue. Reserved concurrency on the function alone is the weaker control: the poller keeps pulling batches, Lambda throttles them, and the messages return to the queue and can reach the dead-letter queue. Batch size and redrive settings shape how messages are grouped and retried, not how many invocations run at once.
• Retries: retry ThrottlingException with exponential backoff and jitter so short bursts succeed rather than failing the message. The SDK's adaptive retry mode adds client-side rate limiting after throttling.

You choose sampling parameters by the task's need for consistency or variety, place them where the API expects them, and prove that a change is an improvement with a live A/B test that keeps each user on one variant.

Temperature, top-p and top-k

For repeatable extraction set temperature at or near 0; this makes output highly consistent but not a guarantee of identical output, so still validate the result. For varied taglines that still follow the brief, raise temperature moderately and widen top-p; lowering top-k narrows choices further. maxTokens controls length and stop sequences control where output ends; neither changes consistency or variety. Many providers advise adjusting temperature or top-p rather than both at once, so check the model's documentation.

Model-specific parameters in the Converse API

Converse normalizes the parameters most models share in inferenceConfig: maxTokens, temperature, topP and stopSequences. Anything specific to a model family, such as Anthropic's top_k, goes in additionalModelRequestFields under the model's native name. additionalModelResponseFieldPaths is the response-side counterpart (extra fields to return), and performanceConfig only selects standard or optimized latency.

inferenceConfig={"temperature": 0.1, "maxTokens": 600},
additionalModelRequestFields={"top_k": 40}

A/B testing on live traffic

An offline evaluation (a Bedrock evaluation job, an LLM judge) compares quality on a dataset before launch; only a live A/B test measures real outcomes such as latency in production, thumbs-up rate or escalation-free resolution. A sound live test needs sticky assignment per user or account (one person must not bounce between variants), an adjustable split, and every metric tagged with the variant.

(Prompt A/B tests with Bedrock Prompt Management and Flows for fairness are covered with Task 3.4.)

You allocate FM resources efficiently by scaling self-hosted endpoints on signals that reflect LLM load and adding capacity ahead of predictable peaks, and you communicate efficiently by reusing SDK clients and their connections instead of rebuilding them per request.

Auto-scaling tuned to GenAI traffic

A model hosted on a SageMaker AI real-time endpoint can take many minutes per new instance to download and load weights, so reactive scaling cannot cover the first minutes of a sharp surge. Combine two policies:

• Scheduled scaling raises the minimum instance count shortly before a known peak (for example 20 minutes before a weekday opening rush) and lowers it afterwards, so instances are loaded when demand arrives.
• Target tracking on concurrency, using the ConcurrentRequestsPerModel metric (or ConcurrentRequestsPerCopy for inference components), follows unpredictable daytime load. Concurrency reflects LLM load better than CPU utilization or invocation counts, because one long generation occupies a model for seconds.

The constraint behind both is model load time: anything that only decides to scale sooner still waits minutes for each new instance, so only capacity added before the surge covers its first minutes, and capacity held permanently at peak pays for idle nights and weekends. SageMaker Serverless Inference does not support GPUs and adds cold starts, so it suits neither LLM hosting nor sharp bursts.

Efficient service communication

• Create SDK clients once. In Lambda, code outside the handler runs once per execution environment, so a client created there keeps its connection pool across warm invocations. A client built inside the handler repeats setup and TLS handshakes on every call. SnapStart and provisioned concurrency address cold starts, not this per-invocation cost.
• Size the connection pool to concurrency. A boto3 client keeps 10 connections by default (max_pool_connections in botocore.config.Config). When more threads share one client than it has pooled connections, say 40 threads, most requests open a fresh connection and logs fill with "Connection pool is full, discarding connection". Raise max_pool_connections to at least the thread count and share the one client. Keep the two problems apart: connection reuse removes per-request setup overhead, while quota and retry settings deal with throttling.
• Keep the network path in perspective. A VPC interface endpoint for the Bedrock runtime changes the network path and keeps traffic private, but it does not shorten model processing time, which is where most seconds go.

import boto3
from botocore.config import Config

# module scope: created once per process / Lambda execution environment
bedrock = boto3.client(
    "bedrock-runtime",
    config=Config(max_pool_connections=40),
)