AIP-C01 RAG Retrieval: Chunking, Embeddings, Vector Search and Query Handling

Retrieval mechanisms are the parts of a retrieval-augmented generation (RAG) system that find the right passages in your own content and hand them to a foundation model (FM) as context. The FM only knows what it was trained on plus what you put in its prompt, so the quality of a RAG answer is capped by the quality of retrieval: if the passage that holds the answer is never found, or is found but ranked too low to reach the prompt, no prompt wording can recover it.

Task 1.5 follows the path a query takes, and each stage has its own AWS tools and its own failure modes:

A useful diagnostic habit runs through the whole topic: first ask whether the right passage is never retrieved (a chunking, embedding, vocabulary or keyword problem), retrieved but ranked too low (a ranking problem that reranking fixes), or retrieved and ranked well but answered badly (a prompt or generation problem). Each symptom has a different fix, and many wrong designs apply a fix from the wrong row.

Designing the vector store itself (metadata frameworks, sharding, multi-index layouts and keeping indexes in sync with changing sources) belongs to Task 1.4 and its vector store lesson. This lesson covers how retrieval uses that store.

Choose the chunking strategy from the structure of the documents and the shape of the questions: split by size when any cut point is acceptable, split by meaning when the text has no structure but changes subject, split into two levels when narrow questions need wide context, and do not split at all when each file is already one self-contained unit. Amazon Bedrock Knowledge Bases offers these strategies on every data source:

Chunk size and overlap

Chunk size is a precision-versus-context trade-off. An embedding is one vector for the whole chunk, so a very large chunk averages many facts together and a narrow question (one torque value, one clause) matches it weakly; it also fills the prompt and can push a RetrieveAndGenerate call past the model's input limit, and the FM may quote a neighbouring fact from the same chunk. A very small chunk matches precisely but can cut an answer in half. Overlap repairs that boundary problem without making chunks large: the text near each boundary is copied into both neighbours, so an answer that crosses a boundary appears whole in at least one chunk. Overlap does not fix a chunk that mixes two subjects; it repeats text and makes prompts slightly longer.

Hierarchical chunking in detail

With hierarchical chunking, the vector search runs over the small child chunks, and each matching child is replaced in the response by its parent chunk. Two consequences matter in practice:

• Fewer results than requested. numberOfResults counts child chunks searched. Children that share a parent collapse into that single parent, so asking for 20 results can return a handful of long passages. That is expected behaviour, not an ingestion fault. (When parsed content holds tables or figures, any chunking strategy can replace a match with the whole page or table and likewise return fewer results.)
• Vector store fit. Large parent text is stored with the vectors. AWS does not recommend hierarchical chunking with an S3 vector bucket at all; large parents (over about 8,000 tokens combined) can also exceed its per-vector metadata limits. A store such as OpenSearch Serverless suits hierarchical chunking.

Both hierarchical levels are cut by size, not by meaning, so hierarchical chunking does not stop a chunk from straddling two subjects, and a parent smaller than a document unit still splits that unit.

Changing chunking later

A data source's chunking configuration is fixed when the data source is created; an update that tries to change it is rejected. To change chunk size or strategy, create a new data source with the new configuration, sync it, and remove the old one.

Use custom chunking when the rule that decides chunk boundaries or chunk metadata cannot be expressed by the built-in strategies, for example one chunk per numbered contract clause, one chunk per table, or a split driven by a document's own heading structure. There are two Lambda patterns inside a Bedrock knowledge base and one outside it:

A custom transformation function is attached to the data source's ingestion configuration and exchanges files with Bedrock through an intermediate S3 bucket: Bedrock writes the content there, the function reads it, transforms it and writes the result back for Bedrock to continue ingesting. If the function does its own splitting, set the chunking strategy to no chunking so Bedrock does not re-split the function's chunks. If the function only adds metadata, keep the native strategy so the function does not have to reimplement chunking.

Keep the levels of metadata apart. A .metadata.json file next to a source document attaches file-level attributes that every chunk of that file inherits, so it cannot say which chapter or page a particular chunk came from. Chunk-level values need the transformation function. Implicit metadata filtering (where an FM builds a filter from the user's question) can only filter on metadata that is already stored; it never creates values. Parsing options, such as a foundation-model or Bedrock Data Automation parser, extract text and structure from complex files but do not decide where chunks are cut; the chunking strategy still does.

Structure-aware hierarchical chunking follows the same rule: when parent and child boundaries must follow headings or sections rather than token counts, write that logic in the transformation function (or pre-split the files) and use no chunking.

Select an embedding model by the language, modality and vocabulary of both the documents and the questions, then pick the smallest vector size and cheapest data type that keeps retrieval accuracy within tolerance on your own data. Every vector in an index must come from the same model with the same settings as the query vectors compared against it.

Dimensions and data type

Vector storage grows with dimensions times bytes per dimension. Moving Titan V2 from 1,024 to 512 float32 dimensions roughly halves storage; to 256, cuts it by about 75 percent. Binary vectors store one bit per dimension instead of 32 bits, a cut of about 97 percent, at the cost of coarser similarity and lower first-pass recall; pairing binary vectors with a reranker restores the ordering of the passages that reach the FM. In a knowledge base, binary vectors are stored only by OpenSearch Serverless and OpenSearch Service managed clusters, so keeping float32 keeps the vector store choice open. Measure the accuracy loss on your own queries before shrinking.

Domain fit and query/document consistency

• Cross-language search needs a multilingual model applied to both sides. An English-only model cannot match a German question to an English passage, and keyword (hybrid) search does not help because the two share almost no words.
• Same model, same settings, both sides. Each embedding model has its own vector space. Questions embedded by Cohere and documents embedded by Titan can have the same dimension and still return meaningless neighbours with no error. Changing the distance metric does not make two spaces comparable.
• Cohere input types. Cohere Embed is trained to pair search_document vectors for stored passages with search_query vectors for questions. Embedding passages as queries (or using classification or clustering, which serve other tasks) lowers retrieval quality.
• Dimension mismatches fail loudly. A query or document vector whose length differs from the index field's dimension is rejected. The only real fix is to embed both sides with the same model and dimension.

Changing the embedding model or dimension

A knowledge base's embedding model configuration is fixed at creation; UpdateKnowledgeBase changes only details such as name, description and service role. An OpenSearch knn_vector field's dimension is also fixed once the index exists. So a new model or dimension means a new knowledge base on a new index (or, in a self-managed pipeline, a new index and a full re-embed). To keep answering during the change, build and sync the new one in parallel, test it, then switch the application to the new knowledge base ID or index.

Evaluate embedding models with a retrieve-only Bedrock evaluation on your own labelled questions, and generate embeddings at scale with batch inference jobs for backfills or batched, quota-aware requests for continuous loads.

Evaluating Bedrock embedding models

Evaluate embedding models by retrieval quality on your own labelled questions, not by public leaderboard rank or by the raw similarity scores the models return. The least-code way on AWS is an Amazon Bedrock retrieve-only knowledge base evaluation job: build one knowledge base per candidate model over the identical documents, run the same prompt dataset against each, and compare the metrics.

• Why not similarity scores? Scores from different models live on different scales; a higher average says nothing about whether the right passage was returned.
• Why not a leaderboard? Public benchmarks measure other corpora and other question styles. Your runbooks, codes and jargon are what matter.
• Why retrieve-only? A retrieve-and-generate evaluation mixes in the generator's behaviour. Retrieve-only isolates the part the embedding model controls.

Retrieve-and-generate evaluations add generation metrics, such as correctness, faithfulness and citation precision and coverage, which describe the generated answer and its citations rather than the retrieval step alone. Evaluate performance characteristics too: vector size and storage cost, maximum input length (which constrains chunk size), multilingual or multimodal support, and throughput limits for the embedding volume you expect.

Generating embeddings at scale

For a large backfill that is not urgent, write the chunks as JSONL records to Amazon S3 and run Amazon Bedrock batch inference jobs with the embedding model: the jobs run asynchronously, avoid per-request throttling and retry code, and batch pricing is lower than on-demand for supported models. Split very large corpora across several jobs, because each job has a record quota.

When embeddings must be produced continuously (new documents every night, or a few at a time), a Lambda function that calls the model on demand is fine, but design it around the model's requests-per-minute quota:

• Send many texts per request where the model allows it. Cohere Embed accepts a list of texts in one request, so batching chunks cuts the request count, which is what the quota limits. Titan Text Embeddings V2 takes one string per request, so it gains nothing from that pattern.
• Do not just add concurrency. Raising reserved concurrency sends requests faster into the same quota.
• Back off and retry on throttling for the on-demand path; a quota increase or Provisioned Throughput adds capacity but keeps per-request calls, and Provisioned Throughput is billed for the commitment, which rarely suits a one-off backfill.
• Set the right input type per side (Cohere search_document for passages) and use the same model and dimension as the index.

Inside a Bedrock knowledge base, the sync job generates embeddings for you; the batching decisions above matter when you run your own pipeline into a self-managed store.

Deploy the vector store whose operating model matches the requirement that dominates: no vector database skills (let the knowledge base quick create one), vectors that must join and commit with relational rows (Aurora PostgreSQL with pgvector), low latency with hybrid search and tuning control (OpenSearch), or a huge, rarely queried corpus where storage cost leads (Amazon S3 Vectors).

Knowledge bases also integrate with other stores, such as MongoDB Atlas, Pinecone, Redis Enterprise Cloud and Amazon Neptune Analytics for graph-based retrieval. Copying relational data into a separate vector store (by streaming changes or by exporting files for a knowledge base to sync) creates copies that lag the source; when filters must reflect the latest transaction, keep the vectors in the database.

Aurora PostgreSQL as a knowledge base vector store

Bedrock reaches Aurora through the RDS Data API and authenticates with a database user whose credentials are stored in AWS Secrets Manager; the knowledge base's storage configuration takes the cluster ARN and the secret ARN. The DBA prepares:

• The vector extension (pgvector) and a table with an ID primary key, an embedding vector column whose dimension matches the model, a chunks text column, a metadata column for Bedrock's source attribution, and optionally a JSONB custom_metadata column that holds all metadata-file attributes.
• An HNSW index on the embedding column with vector_cosine_ops.
• A GIN full-text index on to_tsvector('simple', chunks), which serves the keyword half of hybrid search.
• A GIN index on custom_metadata when that column is used, so metadata filters run efficiently.

Configure OpenSearch vector search by creating a k-NN index whose knn_vector field declares the dimension, the engine, the distance (space type) and the approximate-search method, then tune recall at query time. The settings fall into two groups, and the group decides whether a change needs a reindex:

A minimal mapping looks like this:

PUT /manuals
{
  "settings": { "index.knn": true },
  "mappings": { "properties": {
    "embedding": {
      "type": "knn_vector",
      "dimension": 1024,
      "method": {
        "name": "hnsw",
        "engine": "faiss",
        "space_type": "l2",
        "parameters": { "m": 16, "ef_construction": 128 }
      }
    },
    "text": { "type": "text" }
  } }
}

To diagnose recall, compare approximate results with an exact (brute-force) search on a sample of queries. If recall is short and latency has headroom, raise ef_search first, because it needs no rebuild. Writes whose vectors have the wrong length fail with a mapping error on the vector field, and a mapping's dimension cannot be edited in place, so the fix is always a new index. A Bedrock knowledge base that uses OpenSearch needs the vector, text and metadata fields it is told about at creation; quick create builds them for you.

Hybrid search improves recall for exact strings, and reranking improves the order of what was already retrieved. They fix different symptoms, so the first question is always whether the right passage is missing from the candidates or present but ranked too low.

Hybrid search: keywords plus vectors

Embeddings place similar meanings close together, which is why an error code such as ERR-2093 or a part number can match passages about similar faults instead of the one containing the exact code. Hybrid search adds a keyword (full-text) search over the raw text, so exact identifiers, SKUs, error codes and rare names rank high. In a knowledge base, set overrideSearchType to HYBRID in the Retrieve or RetrieveAndGenerate vector search configuration (SEMANTIC forces vector-only). Hybrid search is available for Aurora PostgreSQL (RDS), OpenSearch Serverless and MongoDB Atlas stores with a filterable text field; S3 Vectors stores are semantic only. Keyword matching cannot bridge languages, so it does not fix cross-language retrieval.

On a self-managed OpenSearch index, run a hybrid query (a keyword sub-query plus a knn sub-query) through a search pipeline with a normalization processor. BM25 scores and vector scores sit on very different scales, so merging raw scores in application code lets one side dominate, and a fixed boost does not help because the BM25 scale changes per query. The processor rescales each sub-query's scores (for example min_max) and then combines them (for example an arithmetic mean with weights) inside OpenSearch, with no extra model call.

Reranking: retrieve wide, keep narrow

A reranker is a model that reads the query and each candidate passage together and scores their relevance more precisely than the vector similarity that produced the candidates. Bedrock offers reranker models such as Amazon Rerank and Cohere Rerank 3.5. Two ways to use them:

• Inside a knowledge base query: add a rerankingConfiguration (type BEDROCK_RERANKING_MODEL, the model ARN and numberOfRerankedResults) to the Retrieve or RetrieveAndGenerate vector search configuration.
• For any candidate list: call the Rerank API on the Agents for Amazon Bedrock runtime with the query and inline text sources, for example 60 candidates from a self-managed OpenSearch index, and keep the top results. The knowledge base rerankingConfiguration cannot point at a store outside a knowledge base.

A reranker can only choose among the passages it receives. If numberOfResults equals numberOfRerankedResults, it merely reorders the same set and the FM sees the same passages. Set numberOfResults well above the number you keep: if the right passage tends to sit around rank 10 and the prompt holds four passages, retrieve 30 to 50 and rerank to 4. Reranking also cannot recover a passage that was never retrieved; that needs better chunking, query handling or hybrid search. Re-embedding candidates with the same embedding model is not reranking, because it repeats the same kind of similarity score.

Transform the query when the user's words are a poor search key: expand it when users and documents use different vocabulary, decompose it when one question needs several retrievals, and rewrite it into a standalone question when it is a conversational follow-up. Each transformation happens before retrieval, because nothing done after retrieval can bring back passages the search missed.

Rules that keep transformation correct

• Search with the transformed query; answer the original. Expanded text is a search key. Send the user's question as written, plus the retrieved passages, to the generation step so the answer responds to what was asked.
• Managed decomposition belongs to RetrieveAndGenerate. The Retrieve API has no query transformation setting. Switching to RetrieveAndGenerate to get it also hands generation to Bedrock, which a team that must keep its own Converse prompt cannot accept; such teams decompose in their own code.
• Managed decomposition stays inside one knowledge base. Routing each sub-question to a different knowledge base needs your own orchestration; Step Functions gives parallelism (Map), declarative retries (Retry) and execution history per iteration with the least custom code, which a hand-written loop or an SQS fan-out does not.
• Do not search with the whole chat history. One long string blurs the embedding across earlier topics and can exceed query length limits. Rewrite to one standalone question instead.
• Decomposition is not expansion. Decomposition splits; it adds no synonyms. Expansion adds words; it does not split.

Fixing vocabulary at query time leaves the index untouched, which is usually less change than rewriting the corpus. More results (numberOfResults), more output tokens, or reranking do not substitute for transformation when the query itself points the search at the wrong place.

Give every consumer the same, well-defined way to retrieve: pick the right Bedrock API for who generates the answer, expose search as a structured tool when a model decides when to search, serve it over the Model Context Protocol (MCP) when many agents or frameworks share it, and put one versioned contract in front of different vector stores.

Bedrock retrieval APIs

Converse (on the Bedrock runtime) sends messages to a model; it does not query a knowledge base on its own.

Vector search as a function-calling tool

With Converse tool use the model decides on each turn whether to search, and the application does the searching. The loop is:

1. Declare a tool in toolConfig with a name, a description and a JSON input schema (for example a query string).
2. When the response's stopReason is tool_use, read the toolUse block, run Retrieve (or any search) with its input. Converse never runs the tool itself; end_turn means the model finished without asking.
3. Append the model's assistant message containing the toolUse block to the conversation, then send a user-role message with a toolResult carrying the same toolUseId and the passages as text or JSON content.

Leaving out the assistant toolUse message puts two user messages back to back with a toolResult that answers nothing, and the request fails validation. Describing a tool in plain prompt text and parsing the reply is the fragile alternative that structured tool use replaces.

MCP clients for vector queries

MCP lets any MCP-capable client (an agent framework such as Strands Agents, or an AI assistant in an IDE) discover a server's tools and their input schemas at runtime, so one search tool serves many frameworks without per-framework wrapper code.

• Shared, hosted tool: put the search behind Amazon Bedrock AgentCore Gateway, which exposes APIs and Lambda functions as MCP tools and authenticates callers (for example with IAM). Agents connect with MCP clients.
• Local developer tool: the open-source AWS Labs Bedrock Knowledge Base Retrieval MCP server runs locally under the developer's AWS profile and discovers only knowledge bases that carry its inclusion tag, so tagging is how a security team limits which knowledge bases are reachable.

A plain REST endpoint or a Bedrock agent is not an MCP server, so an MCP client cannot discover it as a tool without extra work.

A standard retrieval contract across stores

When teams search different stores (knowledge bases, an OpenSearch index, Aurora pgvector) but want to share prompt-building code, define one versioned request and response schema (query, filters, top-k in; passages with text, score and source out) and put an Amazon API Gateway API in front of Lambda adapters, one per store, that translate each backend's native results into that shape. No data moves between stores, and the contract can evolve by version. Letting each team convert native responses in its own code lets shapes drift; consolidating all vectors into one store moves data.