AIP-C01 Troubleshooting GenAI Applications: Context Overflow, Bedrock API Errors, Prompts and Retrieval

Troubleshooting a GenAI application means tracing a bad or failed answer back to the layer that caused it (the content you sent, the API call, the prompt, the retrieval system or the prompt template in production) and fixing that layer rather than the symptom. Task 5.2 of AIP-C01 is built around that discipline: most scenarios describe a symptom with no obvious error, and the right answer is the diagnosis that separates two layers, then the smallest fix that meets the stated constraint.

The habit the exam rewards is isolate before you change. Re-ingesting, raising a parameter or switching models before you know which layer failed adds variables and often hides the cause. Several tools recur across this lesson: Amazon Bedrock model invocation logs (what was actually sent and returned), the Converse response's stopReason (why generation ended), the Retrieve API (what retrieval returned on its own) and AWS X-Ray (where a request spent its time).

Scope: designing evaluation frameworks, LLM-as-a-judge and deployment quality gates belong to the evaluation lessons for Task 5.1; retry and backoff design, X-Ray basics and API Gateway validation to the Bedrock API integration lesson (Task 2.4); choosing a chunking strategy, hybrid search and reranking to the RAG lesson (Task 1.5); prompt approval workflows and regression alarms to the prompt governance lesson (Task 1.6); and standing dashboards to the monitoring lesson (Task 4.3). This lesson uses those features only as troubleshooting tools.

Context window overflow is diagnosed from two distinct signals: a request that is rejected before inference because the input is too long, and a response that stops early because it ran out of room. They have different causes and opposite fixes, so the first job is to tell them apart.

Input too long: rejected before inference

When the prompt (system prompt, conversation history, retrieved passages, tool definitions and the new message) exceeds the model's context window, Amazon Bedrock returns an HTTP 400 ValidationException whose message says the input is too long. It tends to appear intermittently, on the longest documents or the longest sessions, because prompt size varies per request.

To catch this before calling the model, use the Bedrock runtime CountTokens operation. It accepts the same input you would send to Converse (messages and system prompt) or to InvokeModel (the request body) and returns the number of input tokens exactly as that model's tokenizer counts and bills them. It is free of charge and model-specific: token counts differ between models, so count against the model you will call. Not every model supports it on the bedrock-runtime endpoint; AWS documents that some newer Anthropic Claude models are counted through Anthropic's count_tokens API on the bedrock-mantle endpoint instead. A characters-divided-by-four estimate is only approximate (and badly wrong for some languages), and invocation logs or the InputTokenCount CloudWatch metric only describe calls that already happened.

The durable fix is token-budgeted context assembly: decide how many tokens each part of the prompt may use (instructions, history, retrieved context, room for the answer), then fill the context part in priority order until the budget is reached. In a RAG application, add passages in the relevance order the retriever returns and drop the rest; for chat, summarise or drop the oldest turns. This fixes every request, keeps the best evidence and never cuts a passage in half. Cutting the assembled string at a fixed character length is the anti-pattern: it can sever a passage or drop the question itself. Lowering maxTokens does not help either, because it limits output, not input.

A common trigger is a retrieval change, for example raising numberOfResults or adding a data source with large hierarchical parent chunks. Remember that a knowledge base data source's chunking configuration cannot be changed after the data source is created (you would create a new data source), and changing chunking does not bound prompt size anyway; the budget in your application does.

Output stopped early: read stopReason

Every Converse response carries a stopReason that says why generation ended. It is the first thing to check when output looks truncated or malformed.

The distinction between max_tokens and model_context_window_exceeded is the heart of truncation analysis. Output cut off while the input sits well inside the window is the output cap. When the response says the window was exceeded, the problem is the input, whatever maxTokens is set to: any remedy that keeps the same input (continuing, changing how tokens are delivered, ending generation sooner) still has to fit inside an already full window. Reduce the input, typically by condensing it section by section first.

Inputs larger than the context window are processed with dynamic chunking: split the input at natural structure boundaries into pieces that each fit a measured token budget, process every piece, then combine the results. For summarisation this is the map-reduce (section-then-combine) pattern: summarise each section, then summarise the summaries in a final call.

Three design choices decide whether it works:

• Coverage. If the requirement is that every chapter is reflected, every part of the input must reach a model. Keeping only the first part that fits, or the beginning and the end, silently drops content (the classic symptom is a summary that covers only part of the document). Retrieval (for example RetrieveAndGenerate over a knowledge base) selects a handful of relevant chunks for a question; it is the right tool for answering questions, not for guaranteeing complete coverage of a document.
• Boundaries. Split at headings, then paragraph breaks, and subdivide an oversized section at its next boundary. Fixed character or token cuts land in the middle of tables, lists and sentences and garble them; whole chapters as single pieces can still overflow.
• Measurement in tokens, not characters. Token density varies by language and content. A character budget tuned on English text produces far more tokens for many non-Latin scripts, and code or tables tokenise differently from prose. Measure each piece with the target model's tokenizer (CountTokens where the model supports it) and subdivide anything over the budget. A fixed chars-per-token ratio still fails on the languages it was not tuned for.

The pipeline is simple to run on AWS Lambda for a single document, or with AWS Step Functions (for example a Map state over sections) when documents are large or many. Raising maxTokens never lets a larger document in, because it only governs output length. Choosing chunking strategies for a knowledge base index is a separate decision taught in the RAG lesson for Task 1.5.

Amazon Bedrock API errors are diagnosed by their exception name and HTTP status, which point to the layer at fault, combined with what differs between callers that work and callers that fail.

Comparison is the fastest diagnosis. If other functions in the same account and Region invoke the same model successfully, model access and Regional availability are fine, so a 403 from one new function points to that function's execution role not allowing bedrock:InvokeModel on the model. Throttling would be a 429 and would hit everyone; a malformed body would be a 400.

First use of a model in a new account

Serverless foundation models in commercial Regions are enabled automatically the first time they are invoked; there is no per-model access request to approve. Two prerequisites still apply, and missing either returns AccessDeniedException even when the IAM policy allows bedrock:InvokeModel:

• For models sold through AWS Marketplace, the first invocation creates a subscription, so the caller needs aws-marketplace:Subscribe, aws-marketplace:Unsubscribe and aws-marketplace:ViewSubscriptions (only for that first use; an administrator can do it once for the account).
• Anthropic models require first-time use case details, submitted once per account or organization in the Bedrock console or with the PutUseCaseForModelAccess API (AWS documents that opt-in Regions need it submitted again, and that it does not apply to Anthropic models reached through the bedrock-mantle endpoint). If it is submitted from the AWS Organizations management account, member accounts inherit it.

The Model access page where you request each model is the procedure for AWS GovCloud (US), not commercial Regions. Retries and quota increases cannot create a missing prerequisite.

Inference profile errors

Some newer models cannot be invoked on demand by their base model ID. The call fails with a ValidationException saying invocation of that model ID with on-demand throughput isn't supported and to retry with the ID or ARN of an inference profile. The fix is to pass a cross-Region inference profile ID that includes the model (for example one prefixed us., eu., apac. or global.) as modelId. Other consumption models and capacity changes do not address it, because the error is about how the model is addressed, not how much of it you can use.

Inference profiles also explain a confusing pattern: intermittent AccessDeniedException. A cross-Region profile routes each request to one of several destination Regions. AWS documents that an IAM policy for a profile must allow the inference profile and the foundation model in every Region the profile can route to; you can keep that scoped with the bedrock:InferenceProfileArn condition key so the model is only usable through the profile. A policy that allows the foundation model only in the source Region lets requests served locally succeed and denies those routed elsewhere. The fix is the foundation model resource in each destination Region; changes that leave the model's Regional resources out of the policy do not close that gap.

Request validation means making sure every call matches the shape the API and the model expect before it is sent; response analysis means reconstructing what was actually sent and returned when something goes wrong. Both lean on knowing which API owns the request format.

Request shapes: Converse vs InvokeModel

Only rewriting the body (or moving to Converse) fixes a provider mismatch; anything that leaves the body schema unchanged cannot.

Conversation structure and tool use

Converse validates the conversation history before inference. Messages generally alternate between user and assistant roles (strictness varies by model), and tool use has a strict pairing rule: when the model returns stopReason tool_use with one or more toolUse blocks, the next user message must contain a toolResult block for every toolUseId from that turn. If the model asked for two tools in parallel and you return only one result, the request fails with a ValidationException. Only a toolResult block carrying the matching toolUseId satisfies the pairing; output returned in any other form does not count.

Reconstructing requests from model invocation logs

Amazon Bedrock model invocation logging records the full request and response of each model call, with token counts, to CloudWatch Logs, Amazon S3 or both. For forensics, know its limits:

• Large bodies. In CloudWatch Logs, request and response bodies are inline only up to 100 KB. Larger JSON bodies are written to Amazon S3 only if you configure an S3 location for large data delivery, and the log record then points to the object. With CloudWatch Logs as the only destination and no large-data location, large prompts (for example ones carrying a lot of retrieved context) are missing from the record.
• Correlation. Bedrock fills every field of the record automatically except requestMetadata, which the caller supplies per Converse call as key-value pairs. Put a ticket ID, tenant or prompt version there and you can filter records by it. Retention settings change how long records are kept, not what they contain.
• Coverage. Logging is configured per account and per Region, and its destination log group or bucket must be in the same account and Region. A copy of the application in another Region needs its own configuration delivering to a destination in that Region. Only calls through the bedrock-runtime endpoint (Converse, InvokeModel, their streaming forms and the OpenAI-compatible APIs on that endpoint) are captured; AWS documents that calls through the bedrock-mantle endpoint, such as the Anthropic Messages API, are not. Content-type settings select what is logged within that coverage; they do not extend it.
• CloudTrail is not a prompt log. AWS CloudTrail records who called which API and when, but not prompt or response bodies.

Prompt quality problems are troubleshot by controlled comparison: hold everything except the prompt constant, compare versions on the same inputs, and change one variable at a time. Ad hoc trials on a few live requests, especially while also trying a different model, cannot tell you whether an edit helped.

Version comparison and rollback

Amazon Bedrock Prompt Management stores a prompt as an editable draft plus immutable numbered versions. An application can invoke a specific version by passing its version ARN as the modelId of a Converse call, which makes rollback immediate and low-risk: point the application back to the previous version's ARN. With a version ARN as modelId, the request cannot include inferenceConfig, system, toolConfig or additionalModelRequestFields: they come from the stored version, so each version carries its own model settings. Editing the draft does not change production (the application uses a version), and deleting the active version breaks the calls that reference it.

Once production is safe, find the cause with a side-by-side test:

• Run the candidate and the current version over the same fixed set of representative inputs, including edge cases, so differences come from the prompt and the test is repeatable before every release.
• Use identical model IDs and inference parameters (when you invoke versions by ARN, that means identical settings stored in each version). Changing the model and the wording in one release, sampling live traffic from different days, or raising temperature to "cover more outputs" all add confounders.
• Score the outputs the same way each time (the scoring methods themselves, such as LLM-as-a-judge and Bedrock evaluations, are covered in the Task 5.1 lessons).

Systematic refinement for inconsistent outputs

When the same input produces different answers, there are usually two causes, and each needs its own fix:

Lowering temperature alone leaves the ambiguity, so borderline items can still flip between prompt versions; stripping definitions or raising topP makes it worse. Majority voting over several runs also improves consistency but multiplies calls and cost, so it fails a single-call or cost constraint. Change one sampling parameter at a time: AWS documents that some newer Anthropic Claude models reject requests that set both temperature and topP. Systematic refinement is the loop of hypothesis, one change, the same test set and a recorded result, repeated until the metric moves.

You separate retrieval from generation by running retrieval on its own. In Amazon Bedrock Knowledge Bases, the Retrieve API runs only the retrieval step and returns each chunk with its relevance score and source location, while RetrieveAndGenerate mixes retrieval and generation in one call. Call Retrieve with the failing queries before changing anything:

• If the right passage is missing or ranked low, the problem is retrieval: parsing, chunking, embeddings, metadata filters or the vector index.
• If the right passage is ranked first with a high score but the answer is still wrong, the problem is in the generation prompt, the model's instructions or the way context reaches the model.

Lowering temperature, re-ingesting or raising numberOfResults first changes the system while both stages are still mixed, so you learn nothing about the cause.

Generation prompt templates

RetrieveAndGenerate lets you replace the default generation prompt with a custom template. The template uses placeholders that Bedrock fills at run time:

So when answers collapse to "not found" right after someone rewrote the template, while Retrieve shows healthy results, restore $search_results$. Invocation logs of the generation call would show the rendered prompt without the passages.

Preprocessing defects: parsing and metadata

Some retrieval failures are created at ingestion. Inspecting retrieved chunks shows them:

• Flattened tables. The default parser extracts text and loses table layout, so cells run together and values are separated from their column headers. Knowledge Bases offers advanced parsing with a foundation model or with Amazon Bedrock Data Automation, which converts tables and figures into structured text. The parsing configuration is set on the data source and only applies to content as it is ingested; AWS documents that the chunking and parsing configuration cannot be changed after the data source is created, so the managed path is a data source that uses the advanced parser, followed by a sync. Retrieval-side tuning cannot help here, because it works on text that was already damaged at parsing.
• Missing metadata. Metadata filters only match chunks that received metadata during ingestion. For an Amazon S3 data source, the sidecar file must be named after the full file name, including its extension, plus .metadata.json (for handbook.pdf, handbook.pdf.metadata.json) in the same prefix. A sidecar named handbook.metadata.json is ignored, so the document appears in unfiltered results but never in filtered ones. Rename and sync again.

Embedding problems are diagnosed by checking that queries and documents are vectorised by the same model in the way that model expects; drift is diagnosed by watching retrieval quality signals over time. Both fail silently: no errors, just weaker relevance.

Vectorization mismatches

• Asymmetric input types. Some embedding models ask you to declare the role of the text. Cohere Embed on Amazon Bedrock takes an input_type: search_document for text being indexed and search_query for search queries (plus types such as classification and clustering for other tasks). A shared helper that embeds queries as documents weakens relevance without any error. Fix the query path to use search_query; the document vectors were already correct, so no re-indexing is needed.
• Different models at the same dimension. Vectors from two different embedding models live in different spaces even when both have, say, 768 dimensions. The vector store accepts the query because the dimension matches, but distances no longer reflect meaning and relevance collapses. A configuration change to the query model is the usual culprit.

When the model change is deliberate (for example for multilingual support), the safe migration is: revert queries to the old model to restore relevance now, re-embed the whole corpus with the new model into a new index, then switch query embedding and index at the same moment (an OpenSearch alias swap makes it atomic). Re-embedding in place leaves a mixed index during the migration, and switching to a partly built index returns incomplete results.

Retrieval drift

Drift is a gradual decline as user queries and the corpus move apart, for example a new product line whose terminology older documents never used, or a corpus that grows faster than its coverage of common questions. Two automated signals catch it before users complain:

• Relevance score distribution. Publish each query's top retrieval score as a custom CloudWatch metric and alarm when its distribution (for example a percentile) drops. Weakening matches show up here first.
• Reference-query recall. Run a fixed set of golden queries, each with known correct source documents, on a schedule and track how often those sources are retrieved.

Signals that describe generation or ingestion health, rather than what retrieval returns, do not measure retrieval effectiveness. Remediation then targets the cause: new content or metadata, query rewriting, hybrid search or a different embedding model (the retrieval design choices themselves are in the Task 1.5 lesson; full RAG evaluation is in the Task 5.1 lessons).

Vector search problems usually come from how an approximate index interacts with filters and operators, not from the vectors. Two pgvector cases on Amazon Aurora PostgreSQL (or Amazon RDS for PostgreSQL) illustrate the patterns the exam uses; the same reasoning applies to other approximate nearest neighbour (ANN) stores.

Too few results with a filter

An HNSW index search returns a bounded candidate list, controlled by hnsw.ef_search (default 40). A WHERE tenant_id = … condition is applied after the index scan (post-filtering). For a tenant with little data, perhaps only a handful of the 40 candidates belong to it, so ORDER BY … LIMIT 25 returns far fewer rows than requested even though many match.

• pgvector 0.8 iterative index scans fix this while keeping the HNSW index: SET hnsw.iterative_scan = strict_order (results in exact distance order) or relaxed_order (slightly out of order, better recall and speed), so the scan continues until enough rows pass the filter (bounded by settings such as hnsw.max_scan_tuples).
• Raising hnsw.ef_search was the older workaround: more candidates, more chance of filling the limit, at a latency cost. Lowering it makes the problem worse.
• Index build parameters and the distance metric do not change when the filter runs, so they do not address post-filtering. Partitioning or partial indexes per large tenant are alternatives when filters are very selective.

Sequential scan instead of the index

A pgvector index is built for one distance via its operator class, and the query must order by the matching operator for the planner to use it:

A query that orders by <-> against a vector_cosine_ops index cannot use it, so PostgreSQL scans every row and latency grows with the table; EXPLAIN shows a sequential scan. Rewriting the ORDER BY with <=> restores the index scan and keeps cosine ranking. Index tuning parameters only affect an index scan that is not running; the fix is matching the query operator to the index. Tuning OpenSearch k-NN filtering and index parameters for speed is covered in the performance lesson for Task 4.2.

Prompt maintenance troubleshooting keeps templates working as they evolve: you diagnose prompt confusion from the prompts actually rendered, trace behaviour by prompt version, and validate output format so inconsistencies are caught rather than crashing downstream systems.

Diagnosing prompt confusion with CloudWatch Logs and template tests

Prompt confusion is when the model receives something other than what the author intended: an unfilled variable, instructions mixed into data, or contradictory sections. The evidence is the rendered prompt, and model invocation logs hold it as the logged input body. With logs in CloudWatch Logs, a CloudWatch Logs Insights query over the input body field (for example searching for a literal {{customer_name}}) finds every affected request, and the time range shows when the problem began. These requests succeed, so error metrics and alarms never fire, and neither CloudTrail nor X-Ray records prompt bodies by default.

To stop recurrence, add template tests to the build: render every template with sample values for each variable and fail when any placeholder remains unresolved, a required section is empty or the rendered prompt exceeds its token budget.

Logs also reveal structural confusion. If instructions and untrusted content (a customer email, a document) are joined in one user message without separators, the model may follow requests written inside the content ("mark this case as urgent") instead of the task. The fix is structural: put the instructions in the Converse system field, wrap the content in labelled XML tags such as <email>…</email>, and say that the tagged text is material to process, not instructions. Sampling and length settings do not create that boundary; only the prompt's structure does. (Guardrails and prompt-attack filters for hostile input are covered in the Task 3.1 safety lesson.)

Prompt observability with X-Ray and requestMetadata

A prompt observability pipeline lets you slice latency, errors and token use by prompt version, for example during an A/B test served by one function. Two pieces work together:

Anything set per function, such as a resource tag on the Lambda function, cannot separate two versions served by the same function; per-request values can.

Schema validation for format inconsistencies

JSON that parses is not necessarily usable: a response can be valid JSON, end with end_turn, and still omit a required field such as due_date. Validate every response against a JSON Schema before it reaches downstream services, then handle failures (retry once with the validation error in the prompt, or send the item to a quarantine queue) and publish a CloudWatch metric of failures dimensioned by prompt version so a template change that causes them is visible. Only schema validation detects the missing-field case; checks that confirm the JSON is well formed or that generation finished do not, and prompt wording can reduce failures but never detects them.

Amazon Bedrock structured outputs (a JSON schema supplied in outputConfig.textFormat, or strict tool use) constrain the model to the schema, but only for a response that finishes. AWS documents that a response stopped by the token limit still returns incomplete JSON. So check stopReason before parsing, and if large outputs hit max_tokens while the cap is below the model's maximum, raise maxTokens so the largest expected output fits. Any change that leaves the output cap where it was does not add output budget.