Databricks GenAI Engineer: Retrieval Evaluation, Advanced Chunking and Re-ranking

Recall@k is the share of all relevant chunks for a question that appear in the top k results; precision@k is the share of the top k results that are relevant. Both share the same numerator, the number of relevant chunks inside the top k, and differ only in the denominator.

Worked example

Reviewers labeled five chunks as relevant to a question. The retriever returns eight chunks, and those at ranks 1, 4 and 6 are relevant. Three relevant chunks are in the top 8, so precision@8 = 3 ÷ 8 = 0.375 and recall@8 = 3 ÷ 5 = 0.6. Swapping the denominators is the most common mistake: precision divides by what was returned, recall by what should have been found.

Set k to what the chain actually passes

Always measure at the k the chain passes to the LLM. If a chain passes the top 4 chunks, recall@40 can rise sharply (for example because relevant chunks moved from rank 70 to rank 25) while nothing new reaches the top 4 and answers do not improve. A metric computed over ranks the model never reads cannot explain answer quality.

How the metrics move as k grows

Raising k extends the list without reordering it. Recall's denominator is fixed, so recall can only stay the same or rise. Precision's denominator is k itself, so each extra non-relevant result dilutes it. Recall rising while precision falls as k grows is therefore expected behaviour, not a sign that the retriever got worse; the decision is which k suits the prompt budget.

Rank-aware metrics score where relevant chunks appear in the list, which recall@k and precision@k cannot do: both are set metrics, so any reordering inside the top k leaves them unchanged.

• Mean reciprocal rank (MRR) averages 1 ÷ (rank of the first relevant result) across questions. A first hit at rank 1 scores 1.0, at rank 3 scores 0.33, at rank 4 scores 0.25. It suits cases with one correct answer per question, such as an FAQ bot that passes only the top chunk, but it ignores every relevant result after the first.
• NDCG@k (normalised discounted cumulative gain) sums a gain for each relevant result, discounted by its position, and normalises against the ideal ordering. It accepts graded relevance labels (for example highly relevant, partly relevant, not relevant), so it rewards placing highly relevant chunks above partly relevant ones when several chunks are useful.

Rank-aware metrics matter most when judging a reranker: a reranker reorders a fixed candidate list, so recall over that list is unchanged by construction. Its benefit shows up as higher precision or NDCG over the smaller set actually passed to the LLM. Raw similarity scores are not a quality metric at all: they say how close two vectors are, not whether the result is relevant according to ground truth.

MLflow 3 GenAI evaluation (mlflow.genai.evaluate) provides built-in LLM judges, imported from mlflow.genai.scorers, that score traced requests without hand-computed metrics. Three of them assess retrieval specifically, and the key differences are what each one compares and whether it needs ground truth (expectations) in the evaluation dataset.

Two related judges do not assess retrieved context and are easy to confuse with these: RelevanceToQuery checks whether the response addresses the question, and Correctness checks the response against expected facts (so it also needs expectations). Neither reads the retrieved documents.

The ground-truth requirement drives which judges can run where. On unlabeled production traces, for example a daily sample with no budget for labeling, only RetrievalRelevance and RetrievalGroundedness apply. RetrievalSufficiency and Correctness need an evaluation dataset whose rows carry expected facts.

The retrieval judges read the retrieved documents from the MLflow trace, specifically from spans whose span type is RETRIEVER. If the documents sit on a generic span, the judges report that there is no retrieved context to assess, even though the trace visibly contains the search step.

Framework autologging (for example LangChain retrievers) marks retriever spans automatically. In custom Python code you set the type yourself on the decorator:

import mlflow
from mlflow.entities import SpanType, Document

@mlflow.trace(span_type=SpanType.RETRIEVER)
def search(query: str) -> list[Document]:
    hits = run_vector_search(query)  # your Vector Search call
    return [
        Document(page_content=h["chunk_text"],
                 metadata={"doc_uri": h["doc_uri"], "chunk_id": h["chunk_id"]})
        for h in hits
    ]

The span type is half the contract; the output shape is the other half. A RETRIEVER span should return a list of documents, each with page_content and optional metadata (MLflow looks for keys such as doc_uri and chunk_id). A function that joins five chunks into one string gives the judges nothing to assess per chunk. MLflow has other built-in span types, such as CHAIN, TOOL, LLM, EMBEDDING and RERANKER, but they describe other kinds of steps and the retrieval judges do not read them. Logging chunks as run artifacts or putting them into dataset expectations does not help either.

Judge results localise a bad answer to the stage that caused it, which tells you where to spend effort.

Note the trap in the second row: RetrievalRelevance can pass on chunks that are on-topic yet miss the specific fact, so confirming a retrieval miss needs RetrievalSufficiency with expected facts. And a retrieval-side change (reranking, bigger chunks, more results) cannot fix a problem the judges have already placed in generation, and vice versa.

A retrieval evaluation set is a fixed collection of representative questions, each labeled with what should be retrieved or what a correct answer must contain, used to compare configurations repeatably. Without one, choices such as chunk size are anecdotal; with one, you score every candidate (for example several chunk sizes, or with and without overlap) on the same questions and rerun the comparison whenever the pipeline changes. Before launch there is no production feedback, so the evaluation set comes first.

Where the questions come from

• Golden sets hand-written by subject-matter experts (SMEs): highest quality, highest cost in expert time.
• Synthetic sets generated from the corpus by an LLM, with questions and expected facts, then reviewed by SMEs on a sample. This fits when labels are missing and expert time is scarce. A public benchmark does not substitute: it is off-domain for your documents.
• Logged production questions show real traffic but carry no labels, so they cannot drive judges that need expectations until someone labels them.

Label so the set survives chunking changes

Labels tied to chunk IDs break the moment chunking changes, because the chunks no longer exist. Labeling each question with expected facts and source document IDs keeps the set valid for any chunking: it supports RetrievalSufficiency with expectations and document-level recall, and it needs only one round of expert review.

Keep the set independent of what you test

Generating a separate synthetic set from each configuration's own chunks biases each comparison towards the configuration that produced the questions, and borrowing one configuration's set keeps that bias. Compare every candidate on one fixed set whose expected facts do not depend on any candidate's chunks.

Advanced chunking strategies change what gets embedded, or what gets sent to the LLM, when basic fixed-size or structure-aware splitting leaves a trade-off unresolved.

Parent-child (small-to-big) retrieval

Parent-child chunking resolves the conflict between precise matching (small chunks) and enough context for the answer (large chunks) by using different units for each: small child chunks are embedded and searched, and each carries a parent_id that fetches the larger parent section for the LLM. A practical layout indexes only the children in Vector Search with a parent_id column, and stores each parent once in a Delta table keyed by parent_id. Indexing parents alongside children lets whole sections compete in the similarity search, and copying the parent text into every child row duplicates storage.

Because several children of the same section can rank highly, naive parent lookup returns the same parent repeatedly. Retrieve more children than you need, keep the best-ranked child per parent_id, then fetch the first N distinct parents. Reranking alone reorders children without collapsing them by parent, and smaller children or more overlap make repeats more likely.

Semantic chunking

Semantic chunking places boundaries where the embedding similarity between adjacent sentences drops sharply, so each chunk stays on one subject. It suits text without structural markers, such as call or meeting transcripts that drift between subjects. Size-driven splitters (fixed or recursive) cut wherever the token limit falls instead. Two caveats:

• It costs extra embedding calls during ingestion, so use heading- or structure-based splitting where structure exists (for example HTML manuals with section tags, or documents with numbered clauses) and keep semantic chunking for the unstructured corpus.
• A long single-topic passage becomes one very large chunk, and an embedding model silently truncates input past its token limit, so content near the end is never embedded. Keep the semantic boundaries and sub-split any oversized chunk into token-limited pieces.

Contextual enrichment

Contextual enrichment prepends identifying context, such as the document title, product or model name, section heading or a short summary, to each chunk's text before embedding it. Chunks like "The warranty covers parts for one year" or "Set the dial to position 3 before starting" are near-identical across documents; adding their source makes the vectors themselves different, so similarity search can tell them apart, without changing chunk boundaries. Storing the same context in a separate metadata column helps citations and filters, but does not affect a vector computed from the chunk text. A metadata filter can also separate sources, but only if the chain extracts the filter value from each question at query time; enrichment needs only an ingestion change.

The techniques combine: a chain can embed enriched small children and return their parents, fixing ambiguous chunks and missing context together.

Hybrid search runs keyword (full-text) retrieval and vector retrieval for the same query and merges the two ranked lists with Reciprocal Rank Fusion (RRF), which combines results by their rank positions rather than by raw scores. In Databricks Vector Search you request it with query_type="HYBRID" on similarity_search; the default is ANN (vector only), and a keyword-only FULL_TEXT mode is also available (in Beta at the time of writing).

Embeddings capture meaning, so strings that differ by one character, such as part numbers (PN-7732-R versus PN-7732-P), error codes and policy numbers, produce nearly identical vectors and pure vector search returns lookalikes. Hybrid search adds exact keyword matching for those strings while keeping semantic matching for plain-language questions. Keyword-only search fixes identifiers but breaks natural-language queries; more ANN results or a larger embedding model do not add exact matching.

Hybrid search is a first-stage change: it can bring chunks into the candidate list that pure vector search missed. That makes it the complement to reranking, which only reorders candidates already retrieved.

Re-ranking is a second retrieval stage that rescores a small set of first-stage candidates against the query and reorders them, so that the few chunks passed to the LLM are the most relevant. The first stage (vector, hybrid or keyword search) is optimised for speed and recall across the whole index; the reranker is optimised for accuracy over tens of candidates.

Why two stages

A reranker is typically a cross-encoder: it reads the query and each candidate together and outputs a relevance score. That joint reading is why it is more accurate than comparing precomputed embeddings, and also why it is expensive: nothing can be computed before the query arrives, so every chunk scored costs a model call. Scoring millions of chunks per question is impossible within an interactive latency budget, so a fast first stage narrows the index to tens of candidates and the cross-encoder ranks only those. Query rewriting, deduplication and embedding compression are different steps, not re-ranking.

What a reranker cannot do

A reranker can only reorder the candidates it receives; it cannot add a chunk the first stage missed. Two consequences follow:

• If the needed chunk usually sits deeper in the first-stage list than the candidates handed to the reranker, the reranker cannot surface it, however strong it is. Widen the candidate set (within the reranker's latency allowance) and keep passing only the few chunks the prompt budget allows.
• Recall over the candidate list is unchanged by reranking. Judge a reranker by precision@n or NDCG@n on the n chunks actually passed to the LLM.

Match the fix to the symptom: a chunk missing from the candidates is a first-stage (recall) problem, fixed by hybrid search, enrichment or more candidates; a chunk present but ranked too low for the prompt is an ordering problem, fixed by a reranker.

Databricks Vector Search includes a built-in reranker that you enable by passing reranker=DatabricksReranker(columns_to_rerank=[...]) to the existing similarity_search call, so it needs no new serving endpoint, index or training. It is the least-effort way to fix results that are retrieved but poorly ordered. (Databricks is renaming Vector Search to Databricks AI Search, so import paths in current documentation may change; the concept is unchanged.)

Choosing columns_to_rerank

The reranker reads the listed columns in order and considers only roughly the first 2,000 characters it finds. Put the chunk's own text first, then short, useful fields such as the document title. If a long column comes first, such as one holding a whole page or source document, every candidate that shares it looks the same to the reranker (that column's opening characters), the chunk text is never reached, and the order barely changes. num_results sets how many results come back after reranking.

When it fits and when it does not

A self-hosted cross-encoder on Model Serving is the fallback when the built-in reranker does not meet a workload's latency or throughput needs. Fine-tuning the embedding model is a last resort after chunking, enrichment, hybrid search and reranking have been tried.