AIP-C01 RAG, Retrieval and Agent Evaluation: Bedrock RAG Evaluations, LLM-as-a-Judge, Retrieval Metrics and AgentCore Evaluations
RAG, retrieval and agent evaluation is the practice of scoring a GenAI application from several perspectives at once: whether the retriever returns the right passages quickly, whether the generated answer stays grounded in them, whether an LLM judge and your own experts rate it acceptable, and whether an agent completes tasks with the right tools and sound reasoning. On AWS this means Amazon Bedrock RAG evaluation jobs (retrieve-only and retrieve-and-generate, with built-in and custom LLM-as-a-judge metrics), Bedrock human-based evaluation jobs, your own ranking and latency tests, and Amazon Bedrock AgentCore Evaluations for agents. This lesson covers AIP-C01 skills 5.1.5 to 5.1.7.
On this page10 sections
- What does a multi-perspective evaluation of a RAG or agent application measure?
- How do Amazon Bedrock RAG evaluation jobs work?
- Which built-in RAG metric matches which problem?
- How do you evaluate a RAG pipeline that runs outside Amazon Bedrock?
- How do custom metrics and LLM-as-a-judge work, and how do you make judge scores trustworthy?
- How do you collect human feedback as an evaluation perspective?
- How do you test retrieval quality with relevance scoring and context matching?
- How do you measure retrieval latency?
- How does Amazon Bedrock AgentCore Evaluations score agent performance?
- When do you use online, on-demand, batch and dataset evaluation for agents?
- Choose between retrieve-only and retrieve-and-generate Bedrock RAG evaluation jobs, and run both to localise a fault to retrieval or generation
- Match a reported defect to the built-in RAG metric that isolates it, including faithfulness vs correctness and citation precision vs citation coverage
- Evaluate an external RAG pipeline with bring-your-own inference responses and write custom judge metrics with the right input variables
- Make LLM-as-a-judge scores trustworthy through calibration against human ratings and controls for position and length bias, and collect expert ratings with Bedrock human evaluation jobs
- Test retrieval with recall@k, MRR, source-level context matching and p95 latency under concurrency, and re-tune score thresholds after embedding changes
- Measure agent task completion, tool usage and reasoning quality with AgentCore Evaluations (and from the trace for classic Bedrock Agents), and choose online, on-demand, batch or dataset evaluation
What does a multi-perspective evaluation of a RAG or agent application measure?
A multi-perspective evaluation scores the same GenAI application from several independent angles: the retriever (did the right information come back, fast enough?), the generator (did the answer stay with that information and answer the question?), automated judges that apply a rubric at scale, and human experts whose judgment the automated scores must agree with. For agents, it adds task completion, tool use and reasoning quality across multi-step sessions. AIP-C01 skills 5.1.5 to 5.1.7 cover these perspectives; general output-quality metrics, model comparison, A/B and canary tests, regression gates and reporting belong to the companion lesson on output quality and quality gates.
The reason to separate perspectives is diagnosis. A single end-to-end score says the answers got worse; it does not say whether the knowledge base stopped returning the needed passage, the model ignored a passage it received, or an agent called the wrong tool. Each perspective in this lesson answers one of those questions.
| Perspective | Question it answers | Main AWS tool |
|---|---|---|
| Retrieval quality | Did the needed passages come back, in the top positions, within the latency budget? | Bedrock retrieve-only RAG evaluation; your own recall@k, MRR and latency tests |
| Generation quality | Is the answer grounded in the passages, correct, complete and properly cited? | Bedrock retrieve-and-generate RAG evaluation |
| Company-specific rules | Does the answer follow our rubric (compliance wording, tone, policy)? | Custom LLM-as-a-judge metrics |
| Expert judgment | Would our own specialists accept this answer, or prefer the other model's? | Bedrock human-based evaluation jobs with your own work team |
| Agent behaviour | Did the agent finish the task, use tools correctly and reason soundly? | Amazon Bedrock AgentCore Evaluations |
How do Amazon Bedrock RAG evaluation jobs work?
An Amazon Bedrock RAG evaluation job runs a dataset of prompts against a retrieval-augmented generation source and scores the results with an evaluator model (LLM-as-a-judge) that you choose. There are two job types, and picking the right one is the first decision:
| Retrieve-only | Retrieve-and-generate | |
|---|---|---|
| What it evaluates | The retriever on its own: the passages returned for each prompt | The whole pipeline: the generated answer and the passages behind it |
| Needs a generator model? | No — useful before the answering model is chosen, for tuning chunking or embeddings | Yes (or your own saved answers) |
| Built-in metrics | Context relevance, context coverage | Correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, and the responsible-AI metrics harmfulness, stereotyping and refusal |
| Source | A Bedrock knowledge base, or your own saved retrieval results | A Bedrock knowledge base plus a generator model, or your own saved answers and passages |
The dataset is a JSON Lines file in Amazon S3 holding the prompts (up to 1,000 per job) and, where a metric needs it, ground truth: referenceResponses (the reference answer) and optionally referenceContexts (reference passages). The job writes per-prompt scores and the judge's explanation to S3 and shows a report card in the console.
Run both job types to localise a fault
When answers are wrong and you need to know which stage is to blame, run both job types on the same questions. A retrieve-only job with context coverage (scored against the reference answers) shows whether the passages contained the facts needed. A retrieve-and-generate job with faithfulness and correctness shows whether the model used those passages and got the answer right. Questions with high coverage but wrong answers point at generation (prompt, model, context handling); questions with low coverage point at retrieval (chunking, embeddings, filters, number of results). Putting metrics on the wrong job type is a common design error: context metrics belong to retrieve-only scoring of passages, and faithfulness and citation metrics need a generated answer.
Which built-in RAG metric matches which problem?
Each built-in metric answers one narrow question, and the near-twins are where evaluations go wrong. Choose the metric whose definition matches the defect you observed, not the one whose name sounds closest.
| Metric | What it rates | Stays high even when… |
|---|---|---|
| Context relevance | How relevant the retrieved passages are to the question | …the passages are on-topic but miss the specific fact needed |
| Context coverage | How much of the information in the ground-truth answer the retrieved passages cover (requires reference answers) | …irrelevant passages are also returned |
| Correctness | How accurately the response answers the question (reference answers improve it) | …a true detail came from the model's own knowledge rather than your documents |
| Faithfulness | Whether the response avoids content the retrieved passages do not support (hallucination relative to the context) | …the answer is grounded but only covers part of the question |
| Completeness | Whether the response resolves every aspect of the question | …each part answered is unsupported |
| Citation precision | How many of the cited passages were cited correctly — falls on over-citation, when citations point at passages that do not contain the claim | …a claim has no citation at all |
| Citation coverage | How well the cited passages support the response, and whether citations are missing | …every claim is supported but extra, wrong citations are attached too |
| Refusal | How evasive the response is — declining or deflecting answerable questions | …the answer is on-topic but incomplete |
| Helpfulness, logical coherence | Holistic usefulness; freedom from gaps and contradictions in the reasoning | Helpfulness drops for many different defects, so it cannot tell them apart |
Grounding versus truth
The distinction that matters most in practice is faithfulness versus correctness. If a company requires every statement to come from its own documents, a true detail that no retrieved passage contains is still a defect. Correctness rates it as accurate; faithfulness catches it, because it is judged against the retrieved context rather than against the world. Conversely, a faithful answer can still be wrong if the retrieved passage was wrong — that is a retrieval or content problem.
One metric per problem
When reviewers report several distinct defects, track each one with the metric whose definition (in the table above) matches it, so every defect has its own trend line. A holistic metric such as helpfulness moves for all of them at once and so cannot separate causes.
How do you evaluate a RAG pipeline that runs outside Amazon Bedrock?
You evaluate an external pipeline with bring-your-own inference responses: instead of letting Bedrock call a knowledge base and a model, you supply in the dataset what your own pipeline actually produced, and Bedrock only does the judging. This works for pipelines built on Amazon OpenSearch Service, Aurora PostgreSQL with pgvector, a SageMaker AI endpoint or any other stack, and it evaluates the exact system in production without moving documents or changing code.
- Retrieve-only BYOI: each record carries the prompt and the
retrievedPassagesyour retriever returned. - Retrieve-and-generate BYOI: each record carries the prompt, your pipeline's generated answer and the retrieved passages behind it. The passages are essential — faithfulness and citation metrics are judged against them, so a dataset of answers without passages cannot measure grounding.
- Each record names its source with a
knowledgeBaseIdentifier(a label for your pipeline), and one job evaluates one RAG source.
Comparing two configurations fairly
To compare two chunking strategies, embedding models or prompts, run two jobs, one per configuration, and hold everything else constant: the same questions (all of them in both jobs), the same metrics and the same evaluator model. Any other difference between the runs — a different or smaller question set, another judge, changed metrics — means the score gap can no longer be attributed to the change being tested. Rebuilding the pipeline on a Bedrock knowledge base to make evaluation easier evaluates a different system.
How do custom metrics and LLM-as-a-judge work, and how do you make judge scores trustworthy?
A custom metric is your own judge prompt and rating scale, run by the evaluator model alongside (or instead of) the built-in metrics, so company-specific rules — "always state the warranty period when recommending a repair", a four-level brand-voice rubric — appear in the same evaluation report. Built-in metrics use fixed judge prompts, and only a metric included in the job produces a score: documenting a rule or enforcing it at runtime does not measure it. A job can include up to 10 custom metrics.
Input variables decide what the judge sees
| Variable | Contents | Rule |
|---|---|---|
{{prompt}} | The user question | Recommended in both job types |
{{prediction}} | The generated answer | Required in retrieve-and-generate jobs; does not exist in retrieve-only jobs |
{{context}} | The retrieved passages | Required in retrieve-only jobs |
{{ground_truth}} | The reference answer (referenceResponses) | Optional |
{{reference_contexts}} | Labelled reference passages (referenceContexts) | Optional — and the only way referenceContexts is ever used |
Two consequences follow. A rule about the answer needs {{prediction}}; a custom metric that only sees {{context}} judges the passages, not what the customer was told. And if experts have labelled the exact passages an ideal retriever should return, no built-in metric reads them — context coverage compares against the reference answer text — so the comparison needs a retrieve-only custom metric that puts {{context}} beside {{reference_contexts}}.
Define the rating scale (numeric or string values with descriptions, for example the four rubric levels) so Bedrock can parse each verdict into a score and aggregate it across the dataset.
Making LLM-as-a-judge scores trustworthy
An LLM judge is trustworthy only after you have shown that its scores agree with the people whose judgment it replaces, and after its known biases are controlled. A judge that agrees with itself is not thereby correct.
Calibrate against human ratings
Take a set of answers that experts have already rated with the same rubric, score them with the judge, and measure agreement (exact or within-one-level agreement, or a correlation such as Spearman's or Cohen's kappa). The disagreements show where the rubric wording is ambiguous; refine it and re-measure before the score becomes a release gate. Checks that look only at the judge itself — how consistent it is across runs, how it is sampled, which model plays the judge — say nothing about whether it agrees with your experts; only a comparison with human ratings does.
Control the known biases
| Bias | Symptom | Mitigation |
|---|---|---|
| Position bias | In pairwise judging, the first (or second) answer shown wins far more than half the time | Judge each pair in both orders and count a win only when the verdicts agree (treat disagreement as a tie). Any single fixed ordering leaves the bias in the scores. |
| Length (verbosity) bias | Longer answers win even when the extra text adds nothing | Give explicit criteria and state that length beyond what they need earns no credit |
| Self-preference | A judge favours output from its own model family | Use a different, strong model as the evaluator |
| Noise | Scores vary between runs | Low (zero) temperature, a defined rating scale, a short reasoning step before the verdict |
How do you collect human feedback as an evaluation perspective?
When your own specialists — lawyers, clinicians, underwriters — must make the judgment, use an Amazon Bedrock human-based evaluation job with your own work team. Bedrock distributes the prompts, shows workers a managed rating interface and produces a results report, so nothing has to be built. A job can evaluate up to two sources side by side (two models, or saved responses you bring from any pipeline), you choose how many workers rate each prompt, and you define custom metrics, each with a rating method:
| Rating method | What the worker records | Use it when |
|---|---|---|
| ThumbsUpDown | Acceptable or not, per response | A simple pass/fail per answer is enough |
| IndividualLikertScale | A score on a scale for each response on its own | Absolute quality of each output matters, without comparison |
| ComparisonChoice | Which of two responses is better | Only the winner matters |
| ComparisonLikertScale | A preference between two responses and its strength on a five-point scale | You must tell near-ties from clear wins |
| ComparisonRank | The responses in rank order | Ordering several outputs |
Amazon Augmented AI (A2I) is no longer open to new customers, so it is not the choice for a new account; a fully custom review app (for example, AWS Step Functions with a task token and your own UI) works but is far more effort when a Bedrock human evaluation job already fits.
Combining perspectives
Automated and human views answer different questions and are often run together on the same saved responses: a retrieve-and-generate BYOI job scores grounding (faithfulness) on every answer cheaply, while a human job with a custom metric has your specialists rate a rule only they can judge. Production feedback (thumbs ratings in the app) and annotation workflows for continuous improvement are covered in the companion output-quality lesson.
How do you test retrieval quality with relevance scoring and context matching?
Retrieval quality testing measures, on a set of questions labelled with their relevant chunks or documents, whether the retriever returns those items and how high it ranks them. Offline ranking metrics are deterministic and cheap, and they complement the LLM-judged context metrics.
| Metric | Definition | Answers |
|---|---|---|
| Recall@k (hit rate with one relevant item) | Share of questions (or of relevant items) found in the top k results | Does the needed chunk reach the model at all, given it sees the top k? |
| Precision@k | Share of the top k results that are relevant | How much noise is sent with it? (Capped low when each question has one relevant chunk) |
| MRR (mean reciprocal rank) | Average of 1 ÷ the rank of the first relevant result | How high does the relevant chunk rank? |
| nDCG | Rank-discounted gain with graded relevance | Ranking quality when relevance has levels |
Set k to what the model actually receives. A reranker that reorders the top 20 results and passes only the first 5 to the model can never change recall@20 or precision@20 — the same 20 items are counted before and after. Its value shows up in recall@5 or MRR, which measure whether the relevant chunk rises into the slots the model sees.
Similarity scores are not calibrated
The score attached to each result is a similarity under one particular embedding model and index. Its distribution changes when the embedding model, chunking or similarity function changes, so a cut-off such as "drop results below 0.55" tuned for one model can reject almost everything under another even though ranking is still good. After any such change, re-tune the threshold on labelled data, balancing precision (irrelevant chunks dropped) against recall (relevant chunks kept). Averaging result scores is not a quality measure either: a higher average score does not mean better retrieval.
Verifying that retrieved context matches the expected sources
Context matching verification checks that the passages returned come from the right place — the expected source document, tenant or version — by comparing their source location and metadata, not their text. Every Bedrock Knowledge Bases Retrieve result carries a location (for an S3 data source, the document's S3 URI) and the document's metadata attributes.
- Label at document level. Record which source document holds each answer, then assert that its S3 location appears among the results. That check survives re-chunking; comparing chunk text breaks every time chunk sizes change.
- Verify filters per result. When a knowledge base serves many tenants (or product versions) through a metadata filter, assert that the filtered attribute (for example a
product_versionor region field) in every returned result's metadata equals the value in the request filter. Isolation is a property of each result, so a sampled or averaged score cannot prove it. - Provenance needs provenance fields. Only the fields that record where a passage came from (its location and metadata) can show which document, tenant or version it belongs to; relevance and similarity measures describe how well a passage fits the question, not where it came from.
How do you measure retrieval latency?
Retrieval latency is measured by timing the retrieval call on its own, under the expected concurrency, and reporting the percentile the requirement names (for example p95), separately from generation latency.
- Isolate the stage. Time the
Retrievecall in the test harness separately from theConverseorInvokeModelcall. Model metrics such asInvocationLatency,TimeToFirstTokenand output-token throughput describe generation, not retrieval. - Load like production. Sequential single calls never queue, so they understate the tail; drive the test at the stated number of concurrent requests.
- Report percentiles. An average hides the slow tail that a p95 or p99 requirement is about.
Trading recall against latency and cost
More results, hybrid search and reranking usually raise recall but add latency and cost. To choose a configuration, build a grid (number of results × reranking on/off, and so on), measure recall on the labelled set and p95 latency at peak concurrency for each candidate, discard any that miss either limit, and pick the cheapest survivor.
How does Amazon Bedrock AgentCore Evaluations score agent performance?
Amazon Bedrock AgentCore Evaluations scores agents from their OpenTelemetry traces — the prompts, model calls, tool calls and responses that AgentCore Observability records — using built-in and custom evaluators at three levels: session (a whole multi-turn conversation), trace (one request and its response) and tool (one tool call). It works for agents on AgentCore Runtime and for agents built with frameworks such as Strands Agents or LangGraph that emit the supported telemetry; observability (with CloudWatch Transaction Search) must be enabled so the spans exist.
The official skill also names agents built with the classic Amazon Bedrock Agents service. The same three questions apply — task completed, tools used correctly, reasoning sound — but the evidence is the agent's own trace: with tracing enabled on InvokeAgent, each step returns the orchestration rationale, the action group or knowledge base invoked with its inputs, and the observation it got back. Log those traces with the final responses and score them in your own test harness, or save the answers (with any retrieved passages) and judge them in a Bedrock evaluation job with bring-your-own responses.
| Evaluator | Level | What it judges | Ground truth |
|---|---|---|---|
Builtin.GoalSuccessRate | Session | Whether all of the user's goals were completed by the end of the conversation — the task completion rate | Optional assertions (statements that must be true at the end) |
Builtin.Helpfulness, Builtin.Correctness, Builtin.Faithfulness and similar | Trace | Usefulness, factual accuracy, and whether a response conflicts with the conversation history | Correctness can use an expected_response |
Builtin.ToolSelectionAccuracy | Tool | Whether each call used an appropriate tool for the conversation so far | None — judged from the trace |
Builtin.ToolParameterAccuracy | Tool | Whether the arguments passed in each call were correct (values, formats) | None — judged from the trace |
Trajectory match: TrajectoryExactOrderMatch, TrajectoryInOrderMatch, TrajectoryAnyOrderMatch | Session | Programmatic (deterministic) comparison of the tool-call sequence with an expected one | Required expected_trajectory (tool names) |
Task completion
"Did the agent get everything done in this conversation?" is a session-level question, so the measure is Builtin.GoalSuccessRate; turn-level helpfulness or correctness and tool-level scores cannot answer it. When the business has written down what must be true at the end ("a replacement card was ordered for the right account", "the customer was given the delivery date"), supply those statements as assertions and GoalSuccessRate scores each session against them.
Tool usage effectiveness
Separate the two ways tool use fails. Calling the wrong tool (checking stock levels when the user asked to track a parcel) lowers tool selection accuracy; calling the right tool with bad arguments (a weight in pounds where the tool expects kilograms) lowers tool parameter accuracy. Both need no ground truth, so they run on live traffic. Goal success is an outcome score and cannot separate the causes.
When a policy fixes the order of tools, use a trajectory evaluator, chosen by how strict the policy is:
| Evaluator | Passes when | Use when |
|---|---|---|
| Exact order match | The actual sequence equals the expected one, with no extra calls | The sequence is fully prescribed |
| In-order match | The expected tools appear in that order; extra calls in between are allowed | Order matters but extra lookups are permitted |
| Any-order match | All expected tools are called, in any order; extras allowed | Only coverage of the required tools matters |
Reasoning quality in multi-step workflows
Reasoning quality is assessed with custom evaluators, chosen by whether the rule needs judgment or is mechanical.
| Custom LLM-as-a-judge evaluator | Custom code-based evaluator | |
|---|---|---|
| How it works | Your instructions and rating scale, run by a judge model at a chosen level (session, trace or tool), with placeholders such as {context} for the session's prompts, tool calls and responses | An AWS Lambda function that receives the spans and returns a score |
| Best for | Judgment: does each decision follow from earlier tool results, is the plan sound, is the tone on-brand | Deterministic rules: an output field is valid JSON, every quoted carrier appears in the rates tool's result |
| Results | Can vary slightly; costs judge tokens | Repeatable; no judge-model token cost |
To check that a multi-step agent bases each decision on earlier tool output rather than assumptions, write the rubric into a session-level LLM judge whose instructions include {context}, so it sees the whole chain. The rubric needs judgment and the whole chain of tool results and decisions in view, which is why the level (session) and the placeholder matter more than the choice of judge model.
When do you use online, on-demand, batch and dataset evaluation for agents?
AgentCore Evaluations offers several ways to run the same evaluators; the choice depends on whether you are watching live traffic, checking specific sessions, or comparing periods or versions.
| Mode | What it does | Typical use |
|---|---|---|
| Online evaluation | A configuration that continuously samples live sessions (0.01–100%) and scores them with the chosen evaluators; results go to a CloudWatch Logs group and scores are published as CloudWatch metrics (namespace Bedrock-AgentCore/Evaluations) | Production monitoring with alarms on score drops, with no custom code |
On-demand (Evaluate API) | Scores specific session or trace IDs immediately and returns the results | Validating a fix on a few replayed sessions, investigating a reported issue; nothing left running |
| Batch evaluation | Discovers sessions in a CloudWatch log source for a time window, collects their spans and scores them, returning aggregate results | Baselines and before/after comparisons of real past traffic |
| Dataset evaluation (preview) | Replays scenarios against the agent: predefined multi-turn conversations (with assertions, expected responses or trajectories) or simulated users (judged by assertions only) | Pre-release regression suites |
Rules that decide the scenarios
- No ground truth online. Live sessions have no expected answer or trajectory, so evaluators that use ground-truth placeholders (
{assertions},{expected_response}, an expected trajectory) cannot be used in online evaluation. Use them in on-demand, batch or dataset runs. - Online evaluation cannot score the past. Created today, it samples sessions from now on. To compare last week with this week, run two batch evaluations, one per time window, with the same evaluators — no per-session scripting needed.
- Prefer the managed mode. When one of these modes fits the need, use it rather than custom code that recreates its sampling or scheduling.
- Evaluators in use are protected. A custom evaluator referenced by an enabled online configuration cannot be edited or deleted. To change its rubric, clone it with the new instructions and point the configuration at the clone (
UpdateOnlineEvaluationConfigupdates its evaluator list; some consoles ask you to disable the configuration briefly while its evaluator list changes).
Tip. Task 5.1 items on skills 5.1.5 to 5.1.7 are production scenarios about a RAG assistant or agent whose quality must be measured from a particular angle: localising wrong answers to retrieval or generation, picking the one built-in metric that isolates a reported defect (grounding vs accuracy, over-citation vs missing citations, partial vs evasive answers), scoring an external pipeline without changing it, adding a company rubric as a custom metric with the right input variables, proving a judge agrees with experts or removing its biases, and collecting ratings from the company's own specialists in a new account. Retrieval items turn on choosing k to match what the model receives, thresholds that break after an embedding change, provenance checks that survive re-chunking, and latency measured as a percentile under load. Agent items name an AgentCore evaluator or mode decided by level (session, trace, tool), whether ground truth exists, whether the check is judgment or deterministic, and whether the sessions are live, specific or in the past. Several are Select TWO, and wrong options are real evaluation features applied to the wrong job type, level or mode.
- Retrieve-only jobs score the retriever (context relevance, context coverage — coverage needs reference answers); retrieve-and-generate jobs score answers (faithfulness, correctness, completeness, citations, refusal). Run both on the same questions to tell a retrieval fault from a generation fault.
- Faithfulness is judged against the retrieved passages, so a true but unsupported detail lowers it while correctness stays high.
- Citation precision falls on wrong extra citations; citation coverage falls on unsupported or missing citations.
- External pipelines are evaluated with bring-your-own responses that include the retrieved passages; one RAG source per job, so compare configurations with separate jobs on the same questions, metrics and evaluator.
- Custom metrics need {{prediction}} in retrieve-and-generate jobs and {{context}} in retrieve-only jobs; referenceContexts is read only through {{reference_contexts}} in a custom metric.
- Trust a judge only after measuring agreement with expert ratings; swap pair order for position bias and give explicit criteria for length bias.
- Expert ratings come from Bedrock human-based evaluation jobs with your own work team (ComparisonLikertScale captures preference strength); A2I is closed to new customers.
- Evaluate rankers at the k the model receives (recall at that k, MRR), verify provenance by source location and metadata, measure Retrieve latency alone at p95 under load, and re-tune score thresholds after changing embeddings.
- AgentCore Evaluations: GoalSuccessRate (session, optional assertions) for task completion; ToolSelectionAccuracy and ToolParameterAccuracy need no ground truth; trajectory exact / in-order / any-order are deterministic and need an expected trajectory. Classic Bedrock Agents are judged from their InvokeAgent trace.
- Online evaluation samples live traffic into CloudWatch with no ground truth; on-demand checks specific sessions; batch compares past time windows; a custom evaluator used by an enabled online configuration is changed by cloning it.
Frequently asked questions
What is the difference between faithfulness and correctness in Amazon Bedrock RAG evaluation?
Faithfulness rates whether a generated answer avoids content that the retrieved passages do not support, so it measures grounding in your documents. Correctness rates how accurately the answer responds to the question, so a true detail drawn from the model's own knowledge can score as correct while lowering faithfulness. Use faithfulness when every statement must come from your own content.
When should I use a retrieve-only RAG evaluation job instead of retrieve-and-generate?
Use a retrieve-only Amazon Bedrock RAG evaluation job when you want to score the retriever on its own — for example while tuning chunking or embeddings before the answering model is chosen. It rates the returned passages with context relevance and, given reference answers, context coverage. Use retrieve-and-generate when you need to score the final answers, and run both on the same questions to see which stage causes wrong answers.
Can Amazon Bedrock evaluate a RAG pipeline built on OpenSearch or SageMaker AI?
Yes. A Bedrock RAG evaluation job can use bring-your-own inference responses: the dataset holds each prompt with the passages your pipeline retrieved and, for retrieve-and-generate jobs, the answer it generated. Bedrock then judges them with built-in or custom metrics without your pipeline changing. Each job evaluates one RAG source, so compare two pipelines with two jobs on the same questions, metrics and evaluator model.
What does citation precision measure compared with citation coverage?
Citation precision measures how many of the passages a response cites were cited correctly, so it falls when citations point at passages that do not contain the claim. Citation coverage measures how well the cited passages support the response and whether citations are missing, so it falls when claims are uncited or unsupported. An answer that cites the right passage plus extra wrong ones keeps high coverage but loses precision.
How do I know whether an LLM-as-a-judge metric can be trusted?
Score a set of answers that your experts have already rated with the same rubric, then measure agreement between the judge and the experts, for example exact or within-one-level agreement or a rank correlation. Refine the rubric where they disagree and re-measure. For pairwise judging, also judge each pair in both orders to cancel position bias and tell the judge that extra length earns no credit.
Which metric shows whether a reranker improves retrieval?
Measure at the number of chunks the model actually receives. If a reranker reorders the top 20 results and only the first 5 are sent to the model, recall@20 and precision@20 cannot change because the same 20 items are counted. Recall@5 or mean reciprocal rank, compared before and after reranking, shows whether relevant chunks move into the positions the model sees.
How do I measure an agent's task completion rate on AWS?
Use Amazon Bedrock AgentCore Evaluations with the session-level evaluator Builtin.GoalSuccessRate, which judges whether all of the user's goals were completed in a conversation. For predefined test scenarios you can supply assertions — statements that must be true at the end — and GoalSuccessRate scores each session against them. Run it continuously with online evaluation, or on chosen sessions with on-demand or batch evaluation.
Why can't an online evaluation in AgentCore use expected responses or trajectories?
Online evaluation scores sampled live sessions, and live traffic has no ground truth such as an expected response, assertions or an expected tool sequence. Evaluators that rely on those ground-truth placeholders are therefore limited to on-demand, batch or dataset evaluation, where you supply the expected values. Online, use evaluators that judge from the trace alone, such as GoalSuccessRate without assertions, Helpfulness or ToolSelectionAccuracy.
Sign up free to mark lessons complete, bookmark topics and track your exam readiness.