Evaluating Foundation Model Performance

Two foundation models can both sound fluent yet differ sharply in accuracy, helpfulness, and cost. Evaluation is how you decide which model, or which customized version, is actually good enough for your use case. On the exam you are expected to recognize the right evaluation approach for a scenario rather than compute any score by hand.

There are three broad approaches. Human evaluation, sometimes called human-in-the-loop, has people read outputs and rate them; it is the gold standard when quality is subjective, such as helpfulness or tone. Benchmark datasets are standardized test sets with known expected answers, so you can score many models the same way and compare them consistently. Amazon Bedrock Model Evaluation is a managed capability that runs both automatic metric-based evaluations and human-review workflows so you do not have to build the pipeline yourself.

These approaches are complementary. You might use benchmark datasets and automatic metrics for fast, repeatable scoring during development, then bring in human evaluation to confirm the outputs feel right to real users before launch. Choosing between speed, cost, and judgment quality is the theme running through this topic.

Human evaluation puts people in the loop to judge output quality. Reviewers might rate responses for helpfulness, accuracy, tone, or safety, or compare two models side by side and pick the better one. Use human evaluation when the thing you care about is subjective and hard to score automatically, such as whether a chatbot's answer is genuinely helpful and on-brand. Its strength is judgment; its weaknesses are cost and time, since people are slower and more expensive than an automated metric.

Benchmark datasets are curated, standardized test sets, often with expected answers, that let you measure a model consistently. Because every model faces the same questions, benchmarks make comparison fair and repeatable, and they are cheap to rerun. Public benchmarks cover skills like reading comprehension, reasoning, and question answering. The limitation is that a benchmark only reflects the tasks it contains, and a model can score well on a benchmark yet still fall short on your specific real-world data.

In practice you combine them: benchmarks and automatic metrics give fast, objective signals, while human evaluation catches the subjective quality that numbers miss. Recognize the trade-off the exam highlights between automated scale and human judgment.

Amazon Bedrock Model Evaluation is a managed feature that helps you evaluate, compare, and select foundation models for your use case without building your own evaluation infrastructure. It supports two styles of evaluation that map directly to the approaches you just learned.

The first is automatic evaluation, where Bedrock scores model outputs against built-in metrics on tasks such as summarization, question answering, and text classification, using either curated or your own prompt datasets. This gives you fast, repeatable numbers to compare candidate models. The second is human evaluation, where you set up a workflow for reviewers, either your own team or an AWS-managed team, to rate outputs on criteria you define, such as helpfulness, correctness, or tone.

The value is convenience and consistency: instead of assembling datasets, metrics, and reviewer tooling yourself, you configure a job and Bedrock runs it and reports results. For the exam, connect Amazon Bedrock Model Evaluation to the idea of a managed evaluation service that offers both automatic metric-based scoring and human review. When a scenario asks how to compare foundation models on AWS without building a custom pipeline, this is the service to name.

For generative text tasks, the exam expects you to recognize four metrics and what each one measures. You will not calculate them, but you must match the metric to the task.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) checks how much of a reference summary the generated summary covers, so it is the go-to for summarization. BLEU (Bilingual Evaluation Understudy) compares a translation to reference translations, so it targets translation quality. BERTScore compares meaning using embeddings, catching correct answers phrased differently. LLM-as-a-judge uses a capable model to grade outputs, giving human-like judgment at lower cost and greater scale than human review.

The clearest way to lock in these metrics is to match them to concrete tasks. Imagine you are shipping three features and must pick an evaluation method for each.

• A document summarizer. You want to know how well each generated summary captures a reference summary, so you use ROUGE, the standard metric for summarization quality.
• An English-to-Spanish translator. You compare the model's translation against professional reference translations, so you use BLEU, the standard metric for translation quality.
• A customer-support chatbot. Helpfulness and tone are subjective and have no single correct answer, so you rely on human evaluation, optionally scaled with LLM-as-a-judge to score many conversations cheaply.

Notice the pattern the exam rewards. When there is a reference text and you want to measure surface overlap, reach for ROUGE (summaries) or BLEU (translations). When you care about meaning rather than exact wording, BERTScore fits. When quality is open-ended and subjective, use human evaluation, and use LLM-as-a-judge when you need that kind of judgment at scale. Picking the wrong metric, such as BLEU for a chatbot's helpfulness, is a common distractor, so anchor each metric to its task.

Strong metric scores are necessary but not sufficient. A model can post good ROUGE numbers and still fail to move the outcomes a business cares about, so you also evaluate against business objectives. The exam frames these as productivity, user engagement, and task engineering, meaning the model should help people get more done, keep users interested and returning, and handle the tasks it was designed for.

To judge this, you look beyond the model's raw output to its real-world effect. Does the writing assistant let employees finish drafts faster, raising productivity? Does the recommendation feature increase how often users interact with the product, raising engagement? Does the assistant reliably complete the tasks it was built to do? These questions connect model quality to value.

Note the distinction from Domain 1 metrics like accuracy, precision, recall, and F1, which measure classic predictive models on labeled test data. Those remain useful background, but here the focus is generative output quality plus business impact. When a scenario emphasizes whether an FM delivers value to the organization, think in terms of productivity and engagement, not just a text-similarity score.

Foundation models usually live inside larger applications, and you must evaluate the whole application, not just the raw model. The exam calls out three application types.

• RAG systems: check both retrieval and generation. Did the system fetch relevant, correct documents, and did it produce an accurate answer grounded in them? Poor retrieval sinks answer quality even with a great model.
• Agents: check whether the agent completed the task correctly, used the right tools, and reached the goal, since a fluent response that fails the task is still a failure.
• Workflows: check the end-to-end outcome across multiple steps, not just any single model call.

To tie evaluation to value, the exam lists business-objective alignment metrics. Task completion rate measures how often the application successfully finishes what the user asked. User satisfaction captures how happy users are, often via ratings or feedback. Cost per interaction tracks how much each request costs, which matters because token-based pricing scales with usage. Together these link a model's quality to real outcomes, so a good evaluation blends generative metrics, human judgment, and these alignment measures.