Evaluating Foundation Model Performance
Evaluating a foundation model (FM) means measuring whether its outputs are good enough for your goal, using the right approach and metric for the task. The three main approaches are human evaluation, where people judge quality; benchmark datasets, which are standardized test sets; and Amazon Bedrock Model Evaluation, a managed service that runs automatic, judge-model, human and RAG evaluations for you. For generative text you should recognize ROUGE for summaries, BLEU for translations, BERTScore for meaning-level similarity, and LLM-as-a-judge, where another model grades the output, and know which of them need reference texts. Good evaluation does not stop at scores: you also check whether the FM meets business objectives, evaluate whole applications such as RAG systems and agents step by step, and track alignment metrics like task completion rate, user satisfaction and cost per interaction.
- Describe evaluation approaches: human-in-the-loop evaluation, benchmark datasets, and Amazon Bedrock Model Evaluation, with their trade-offs.
- Distinguish Amazon Bedrock automatic, human-worker, judge-model and RAG evaluation jobs, including their datasets and built-in metrics.
- Define ROUGE, BLEU, BERTScore and LLM-as-a-judge, and identify which metrics need reference texts.
- Choose an appropriate metric or method for summarization, translation and open-ended chat scenarios under data and budget constraints.
- Determine whether an FM meets business objectives such as productivity, user engagement and task engineering.
- Evaluate RAG systems, agents and workflows in a way that locates where they fail.
- Interpret business-objective alignment metrics: task completion rate, user satisfaction and cost per interaction.
Why and how you evaluate a foundation model
You evaluate a foundation model to find out whether its outputs are good enough for your use case, because two models can both sound fluent yet differ sharply in accuracy, helpfulness and cost. Evaluation is how you decide which model, or which customized version, to ship. The exam expects you to recognize the right evaluation approach for a scenario rather than compute any score by hand.
There are three broad approaches. Human evaluation, also called human-in-the-loop evaluation, has people read outputs and rate them; it is the gold standard when quality is subjective, such as helpfulness or tone. Benchmark datasets are standardized test sets with known expected answers, so you can score many models the same way and compare them consistently. Amazon Bedrock Model Evaluation is a managed capability that runs automatic, judge-model and human-review evaluations so you do not have to build the pipeline yourself.
These approaches are complementary. You might use benchmark datasets and automatic metrics for fast, repeatable scoring during development, then bring in human evaluation to confirm the outputs feel right to real users before launch. Choosing between speed, cost and quality of judgment is the theme running through this topic.
Human evaluation and benchmark datasets
Human evaluation puts people in the loop to judge output quality. Reviewers might rate responses for helpfulness, accuracy, tone or safety, or compare two models side by side and pick the better one. Use it when the quality you care about is subjective, or when the decision itself must be made by people, such as clinicians deciding whether patient-facing discharge instructions are clear enough to send. Its strength is judgment; its weakness is that every rating needs a person, so it is slow and expensive and does not scale to large volumes of output.
Benchmark datasets are curated, standardized test sets, usually with expected answers, that let you measure a model consistently. Because every model faces the same questions, benchmarks make comparison fair and repeatable, and they are cheap to rerun when a new model version is released. Public benchmarks cover skills like reading comprehension, reasoning and question answering. The limitation is that a benchmark only reflects the tasks it contains: a model can score well on a public benchmark and still fall short on your own data and prompts.
In practice you combine them: benchmarks and automatic metrics give fast, objective signals, while human evaluation catches the subjective quality that numbers miss.
Amazon Bedrock Model Evaluation
Amazon Bedrock Model Evaluation (Amazon Bedrock evaluations) is a managed feature for evaluating, comparing and selecting models and RAG sources without building your own scoring or review infrastructure. You configure an evaluation job, Bedrock runs it and produces a report. There are four kinds of job.
| Job type | Who or what scores | Dataset | Notes |
|---|---|---|---|
| Automatic (programmatic) | Built-in metrics: accuracy, robustness and toxicity | Built-in curated datasets or your own prompt dataset | Task types: general text generation, summarization, question and answer, text classification. Evaluates one model per job. |
| Human workers | Your own work team (employees or subject-matter experts) rating on criteria you define | Your own prompt dataset, stored in Amazon S3 | Can rate or compare the responses of up to two models in one job. |
| Judge model (LLM-as-a-judge) | A second LLM scores each response and explains the score | Your own prompt dataset | Human-like judgment at scale, no reviewers needed. Can also score inference responses you supply from a model outside Bedrock. |
| RAG evaluation | An LLM scores a knowledge base or other RAG source | Your prompts plus ground truth: expected retrieved texts and responses | Two types: retrieve only and retrieve and generate. Can also score responses you supply from an external RAG system. |
Two details matter when a scenario hinges on them. First, the built-in accuracy metric compares each output with a ground-truth reference response (for example, real-world-knowledge, BERTScore or F1 scores depending on the task), so it is useless when you have no reference answers; robustness checks how much output changes when the prompt is slightly perturbed, and toxicity is scored by a classifier. Second, a retrieve-only RAG evaluation scores just the passages the knowledge base returns, so it isolates retrieval (for example, after you switch the embedding model or add a new batch of source documents) without the response model's wording affecting the result, while retrieve and generate also scores the generated answer.
When a scenario asks how to compare foundation models on AWS with automatic scores and human ratings without building a custom pipeline, Amazon Bedrock Model Evaluation is the service to name. It is distinct from Amazon Bedrock Knowledge Bases, which supply retrieval for RAG, and Amazon Bedrock Guardrails, which filter content; neither of those scores or compares models.
Generative evaluation metrics
For generative text, the exam expects you to recognize four metrics and match each to the task it measures; you will not calculate them.
| Metric | How it scores | Needs a reference text? | Typical use |
|---|---|---|---|
| ROUGE | Recall-oriented word and n-gram overlap: how much of the reference the output covers | Yes | Summarization |
| BLEU | Precision-oriented n-gram overlap: how much of the output appears in the references, with a penalty for outputs that are too short | Yes | Machine translation |
| BERTScore | Similarity of meaning between output and reference, using embeddings | Yes | Tasks where correct answers may be worded differently |
| LLM-as-a-judge | A capable model grades the output against a rubric you write | No | Scalable scoring of open-ended text |
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) asks how much of a reference summary's words and phrases the generated summary covers, which is why it is the summarization metric. BLEU (Bilingual Evaluation Understudy) was designed for machine translation and asks how precisely the output's word sequences match one or more reference translations. Both count shared wording, so like an exact-match score they penalize a correct answer phrased differently. BERTScore still compares against the same references but by meaning, so a correct paraphrase is credited without writing new references. LLM-as-a-judge sends each output and a rubric (for example, accuracy and helpfulness) to a second model and records its score; it gives human-like judgment at far lower cost and greater scale than human review.
Which signals work without reference texts
ROUGE, BLEU, BERTScore and Bedrock's built-in accuracy metric all need a reference or ground-truth answer for every item. Human evaluation, LLM-as-a-judge and user satisfaction ratings do not: they judge outputs against criteria or record users' opinions. Classifier-based scores such as toxicity also need no reference. So for a large volume of open-ended output with no reference answers, such as model-written product descriptions, the overlap metrics cannot be computed until someone writes references, while a judge model or human reviewers can score the output straight away.
Choosing the right metric: a worked example
The right metric follows from the task and from what data you hold. Imagine a hospital group shipping three features.
- A clinical-visit recap. Doctors have written reference summaries for a sample of visits, and the question is how much of each reference the generated recap captures. Coverage of a reference summary is what ROUGE measures.
- Patient leaflets localized from English into Portuguese. Professional reference translations exist, so BLEU, the metric built for translation, fits. Because BLEU rewards matching word sequences, outputs that use valid synonyms or a different sentence order score low even when they are correct; BERTScore compares meaning against the references you already have, so it credits those outputs without commissioning new references.
- A symptom-triage assistant. Whether a reply is safe, clear and kind has no single correct wording. Clinicians reviewing outputs (human evaluation) give the most trusted judgment and are the right choice when people must make the decision; LLM-as-a-judge applies a written rubric when the volume is far beyond what reviewers can read.
The pattern: reference text plus surface overlap points to ROUGE (summaries) or BLEU (translations); meaning rather than wording points to BERTScore; open-ended or subjective quality points to human evaluation or LLM-as-a-judge. Mixing these up weakens the signal. A translation metric applied to summaries rewards precise wording but not coverage of the reference, and using BLEU or ROUGE to judge helpfulness fails because those metrics only count wording shared with a reference, and helpfulness has no single reference.
Does the FM meet your business objectives?
An FM meets business objectives when it changes the outcomes the business cares about, not merely when it posts good text scores. A model can hit its ROUGE target and still fail to deliver value, so you also evaluate against the objectives the exam names: productivity, user engagement and task engineering. In plain terms, the model should help people get more done, keep users coming back, and reliably handle the tasks it was designed for.
To judge this, measure real-world effect rather than output quality. Does a contract-review assistant cut the hours lawyers spend per contract (productivity)? Do they keep choosing it over their old process month after month (engagement)? Does the assistant complete the tasks it was built for (task engineering)? Benchmark rank and text-similarity scores answer a different question: how capable the model is in general, not whether this deployment helps this business.
Note the distinction from Domain 1 metrics like accuracy, precision, recall and F1, which measure classic predictive models on labeled test data. Here the focus is generative output quality plus business impact.
Evaluating applications and alignment metrics
You evaluate an application built on an FM by testing the whole system end to end, not just the raw model, because most failures come from how the model is wired to data and tools. The exam names three application types.
- RAG systems: score retrieval relevance (did the system fetch passages that contain what the question needs?) and answer grounding, also called faithfulness (does the answer stick to those passages, without invented details?) as two separate measures. Separating them locates the failure: if relevant passages are retrieved but answers still include facts found in none of them, the problem is in generation, and better retrieval will not fix it; if the passages are irrelevant, fix retrieval first. A strong benchmark score for the base model says nothing about either step.
- Agents: check whether the agent completed the task correctly, and also whether at each step it chose the right tool and passed the right values to it. Checking each tool call lets you find the step that broke in a multi-step sequence, because fluent individual replies do not prove that the end result is correct.
- Workflows: check the end-to-end outcome across all the steps, not just any single model call.
Business-objective alignment metrics
| Metric | What it measures | Example signal |
|---|---|---|
| Task completion rate | How often the application fully finishes what the user asked | Share of travel-booking conversations that end in a confirmed booking |
| User satisfaction | How users feel about the help they got | Star ratings, thumbs up or down, survey answers, would-use-again feedback |
| Cost per interaction | What each request costs to serve | Token and infrastructure spend divided by requests |
The three move independently. High completion and falling cost do not mean users are happy; when the concern is how users feel about the experience, user satisfaction is the only one of the three that measures it, before or after a change. Cost per interaction follows from token-based pricing, where input and output tokens are usually priced separately and output tokens typically cost more: each request costs roughly (input tokens × input price) + (output tokens × output price), plus any infrastructure cost. A lower list price per token therefore does not guarantee a cheaper request, because prompt size and answer length drive the token count; compare models on the cost of full requests over representative traffic, not on the price list. Latency is a separate measure of speed, not spend.
Tip. Expect scenario questions that give a task, the data available and a constraint, and ask for the evaluation method. Coverage of a reference summary points to ROUGE, machine translation against reference translations to BLEU, correct-but-reworded outputs scored against existing references to BERTScore, and rubric scoring at scale with no reference answers or reviewers to LLM-as-a-judge; people's own sign-off on subjective quality points to human evaluation, and fair repeatable comparison on a standard question set to benchmark datasets. Know which metrics need reference texts, the Amazon Bedrock Model Evaluation job types (automatic built-in metrics, human workers, judge model, RAG retrieve only versus retrieve and generate) and their dataset rules. For applications, separate RAG retrieval relevance from answer grounding, judge agents by tool choice, tool inputs and end-to-end outcome, and tell business objectives (productivity, engagement) and alignment metrics (task completion rate, user satisfaction, cost per interaction as input and output tokens times their prices) apart from text-quality scores.
- Human evaluation suits subjective quality but is slow and costly; benchmark datasets give fair, repeatable comparison but may not reflect your own data; Amazon Bedrock Model Evaluation runs evaluations as managed jobs.
- Bedrock automatic jobs use built-in metrics (accuracy, robustness, toxicity) on built-in or your own datasets for one model; human jobs need your own prompt dataset and can compare up to two models; judge-model and RAG jobs score with an LLM.
- Bedrock's built-in accuracy metric compares outputs with ground-truth reference responses; a retrieve-only RAG evaluation scores retrieval alone, while retrieve and generate also scores the answer.
- ROUGE is recall-oriented overlap with a reference summary; BLEU is precision-oriented n-gram overlap with reference translations; both, like exact match, penalize correct paraphrases.
- BERTScore compares meaning against the same references; LLM-as-a-judge grades against a rubric and needs no reference text, as do human evaluation and user ratings.
- Beyond metrics, judge whether the FM meets business objectives like productivity, user engagement and task engineering.
- Evaluate RAG retrieval relevance and answer grounding separately, and agents by tool choice and tool inputs as well as task completion.
- Cost per interaction is (input tokens x input price) + (output tokens x output price) per request, so a model with a lower per-token price but longer answers can cost more.
Frequently asked questions
What is ROUGE used for, and how does it differ from BLEU?
ROUGE is used to evaluate summarization. Both compare generated text with reference text, but from opposite directions: ROUGE is recall-oriented and measures how much of a reference summary the output covers, while BLEU is precision-oriented and measures how much of the output's word sequences appear in reference translations, with a penalty for outputs that are too short, so BLEU is used for machine translation.
Which evaluation metrics need reference answers?
ROUGE, BLEU, BERTScore and Amazon Bedrock's built-in accuracy metric all need a reference or ground-truth answer for every item they score. Human evaluation, LLM-as-a-judge and user satisfaction ratings do not, because they judge outputs against criteria or record users' opinions; classifier-based scores such as toxicity also need no reference.
What is BERTScore and when should you use it instead of BLEU or ROUGE?
BERTScore compares a generated text with a reference by meaning, using embeddings, rather than by counting shared words. Use it when correct outputs are often worded differently from the reference, because BLEU, ROUGE and exact-match scoring mark those correct paraphrases down while BERTScore credits them using the same references.
What is LLM-as-a-judge?
LLM-as-a-judge is an evaluation method in which a capable model grades another model's outputs against a rubric you define, such as accuracy or helpfulness, and returns a score. It needs no reference answers and gives human-like judgment at far greater scale and lower cost than human review, which makes it useful for scoring large volumes of open-ended text.
What evaluation jobs does Amazon Bedrock Model Evaluation offer?
Amazon Bedrock Model Evaluation offers automatic (programmatic) jobs that score one model with built-in metrics for accuracy, robustness and toxicity on built-in or your own datasets; human-worker jobs in which your own team rates up to two models on your own prompt dataset; judge-model jobs in which a second LLM scores responses; and RAG evaluations of knowledge bases, either retrieve only or retrieve and generate.
When should you use human evaluation instead of an automatic metric?
Use human evaluation when the quality you care about is subjective and has no single correct answer, such as whether a chatbot's reply is helpful and on-brand, or when people must make the final call. Automatic overlap metrics need reference texts and only count shared wording. Human evaluation captures nuanced judgment but is slow and costly because every rating needs a person.
How do you evaluate an application built on a foundation model?
Evaluate the whole application, not just the model. For a retrieval augmented generation (RAG) system, score retrieval relevance (do the retrieved passages contain what the question needs?) and answer grounding or faithfulness (does the answer stick to those passages?) separately, so you can tell a retrieval failure from invented details in generation. For an agent, check tool choice and tool inputs at each step as well as task completion; for a workflow, check the end-to-end outcome.
What is cost per interaction and how is it calculated?
Cost per interaction is a business-alignment metric for what each request to an AI application costs to serve. For token-priced models it is roughly input tokens times the input price plus output tokens times the output price, plus any infrastructure cost; output tokens usually cost more. Because prompt size and answer length drive token use, a model with a lower per-token price can still cost more per interaction.
Source
This lesson covers the "Applications of Foundation Models" domain of the official AIF-C01 exam guide. Vendors revise their guides — check the source for the current version.
- AWS Certified AI Practitioner (AIF-C01) exam guide — Amazon Web Services
Sign up free to mark lessons complete, bookmark topics and track your exam readiness.