AIP-C01 GenAI Evaluation: Output Quality Metrics, Bedrock Evaluations, A/B and Canary Testing, Quality Gates

A foundation model (FM) produces open-ended text, so there is rarely one correct output to compare against, and quality has several independent dimensions. A classifier is scored with accuracy or F1 against a fixed label; an FM answer can be worded a hundred valid ways, can read fluently while being wrong, and can contradict itself across requests. An assessment framework for GenAI therefore scores meaning rather than surface tokens and scores each quality dimension separately.

The four core dimensions

Lexical versus semantic scoring

Lexical metrics count shared words: exact match, ROUGE (n-gram recall, common for summaries) and BLEU (n-gram precision). They penalise a correct answer that is simply worded differently from the reference. Semantic metrics compare meaning: BERTScore embeds both texts with a transformer and matches tokens in embedding space, so synonyms and reworded sentences still score well; cosine similarity of sentence embeddings does the same at the whole-answer level. When failures turn out to be valid paraphrases, the fix is to switch to a semantic metric, not to normalise harder. Perplexity measures how predictable the text is to a language model; it needs no reference and says nothing about correctness.

Gate each dimension, not an average

A single averaged score lets one dimension compensate for another: a release can become more fluent and less accurate and still clear an average floor. Give each dimension its own minimum (and the critical ones, such as factual accuracy in finance or health, a tighter one). Any scheme that folds the dimensions into one number before gating has the same weakness, whatever the weights, and it also hides which dimension moved.

Consistency is its own test

Per-answer scores judge each answer in isolation, so two contradictory answers to the same underlying question can both score high on relevance, accuracy and fluency. Measure consistency explicitly and without references by building paraphrase groups (several wordings of each test question) and scoring the share of groups whose answers reach the same decision. Asking the identical question several times measures a different property, sampling stability at the production temperature; both are useful, but they detect different failures.

Amazon Bedrock Model Evaluations runs managed evaluation jobs that score one or more models on a dataset and produce a report card in the console plus result files in Amazon S3. There are three families of model evaluation job; choose by who (or what) does the scoring.

Bedrock also evaluates knowledge bases (retrieval and retrieve-and-generate). RAG, retrieval and agent evaluation, and the design of LLM-as-a-judge prompts and rubrics, are covered in the lesson on RAG, retrieval and agent evaluation; in this lesson judge jobs appear as the engine for continuous scoring and update validation.

Automatic (programmatic) jobs

• One task type per job: general text generation, text summarization, question and answer, or text classification. Comparing several kinds of task means several jobs.
• Built-in metrics, each computed in a task-appropriate way: accuracy and robustness for every task type; toxicity for general text generation, summarization and question and answer (text classification has no toxicity metric):
  • Accuracy compares the response with the reference (for example BERTScore for summarization, F1 for question and answer, classification accuracy for classification).
  • Robustness perturbs each prompt (typos, case changes, whitespace and similar small edits) and reports how much the score changes between the original and perturbed prompt (for example deltaF1 for question and answer). It is the metric for "does quality hold up when users type messily?"
  • Toxicity scores each response with a toxicity detector (not offered for text classification).
• Datasets: built-in curated datasets (sampled to 100 prompts) or a custom dataset in S3 of up to 1,000 prompts, as JSONL with prompt, referenceResponse and an optional category. When category is present, the report shows each metric per category as well as overall, so one job can break results down by document type, product area or language. S3 object tags or separate prefixes do not do this.

Bringing your own inference responses

Judge-based and human-based jobs can score responses you generated elsewhere instead of invoking a Bedrock model: the dataset records carry the precomputed response with a modelIdentifier naming its source. This lets one job compare a Bedrock model with a model on an Amazon SageMaker AI endpoint, another cloud or an on-premises server without migrating it, and it is what makes scoring logged production responses possible.

A human-based evaluation job shows prompts and responses to human workers in an AWS-provided rating portal and records their ratings per metric, so you build no labeling UI. It compares up to two sources per job (two Bedrock models, or a Bedrock model and your own inference responses), with metrics you define and a rating method per metric.

Rating methods

Work team and setup

• Bring your own work team: your employees or experts, managed as a private work team (Amazon Cognito user pool) so sensitive data stays with people you choose. You create and manage the team (and invite its workers) from the Bedrock console.
• Console-created jobs need a CORS configuration on the S3 bucket that holds the dataset and output, so the rating portal can read and write it.
• API-created jobs (CreateEvaluationJob, for example from a release pipeline) need a SageMaker AI flow definition whose AwsManagedHumanLoopRequestSource is AWS/Bedrock/Evaluation, referencing your private work team ARN and the evaluation task UI; you pass its ARN to the job. The Bedrock service role must be able to access the output bucket named in the flow definition.

Human jobs work on a curated dataset (the same 1,000-prompt cap applies to custom datasets). They are an evaluation tool for comparing models, not a general labeling service for thousands of production answers; that is what annotation workflows are for.

You choose a configuration by first discarding every candidate that fails a hard requirement, then optimising the stated objective among the rest. Multi-model evaluation produces the inputs: run the same evaluation set through each candidate and record quality scores, latency percentiles and token counts.

Token efficiency and cost per request

Cost per request = input tokens × input price + output tokens × output price. Measure the token counts on your own prompts: a model with lower per-token prices can cost more per request if it writes longer answers or emits reasoning tokens, which are billed as output tokens. Token efficiency is the quality delivered per token spent.

Cost per successful outcome

When the goal is a business result (a resolved dispute, a completed booking), divide cost per request by the success rate: cost per success = cost per request ÷ success rate. A cheaper, weaker model can lose on this measure because more of its requests produce nothing useful. For example, a model costing $0.012 per request with a 90% success rate costs about $0.0133 per success, while a $0.009 model that succeeds 60% of the time costs $0.015 per success: the cheaper request is the dearer outcome.

Latency-to-quality trade-offs

Latency matters as a threshold (for example a p95 response time under 2 seconds for an interactive assistant), not as something to minimise beyond it. Once a candidate meets the latency and quality floors, extra speed or extra quality headroom does not outweigh a stated objective such as lowest cost. Use percentiles (p90, p99) rather than averages, because users notice the tail. Present the trade-off as a scatter of quality against cost or latency per candidate, with the thresholds drawn in, so the feasible set is obvious.

Business outcomes decide

Offline scores are proxies. When an online test has a stated business goal (for example tickets resolved without follow-up, or completed checkouts), that outcome measured over every user in each arm decides the test. Proxy signals can disagree with it and should not override it: ratings come from the minority of users who choose to rate and are biased, and an evaluator's correctness score can rise while answers become harder to act on.

Online testing exposes candidate models to real traffic in three distinct ways: an A/B test splits users between models to measure outcomes, a canary releases a new version to a small share with automatic rollback, and a shadow test runs the candidate on copies of live requests that users never see.

A/B testing: keep each user in one arm

A conversation that switches models between turns spoils the comparison, so assignment must be per user, not per request.

• SageMaker AI endpoints: production variant weights route each request randomly. For a user-sticky test, hash the user ID into a bucket in the application and pass the chosen variant in the TargetVariant parameter of InvokeEndpoint. Sticky session routing exists, but it pins a session to an instance, not a user to a variant for the life of the test.
• Amazon Bedrock: the split lives in your code. Keep the model IDs and the challenger percentage in an AWS AppConfig feature flag that the function reads, and assign each user by comparing a stable hash of the user ID with the configured percentage. The percentage can change, or the test can end, within minutes with no redeploy.
• Lambda alias weights and API Gateway stage canaries split per request and need a deployment to change; they are release mechanisms, not experiment assignment. Bedrock intelligent prompt routing picks between models of one provider family to balance cost and quality; it is not an experiment mechanism.

Canary releases with automatic rollback

• SageMaker AI deployment guardrails: a blue/green update with CANARY traffic routing shifts a canary share of capacity to the new fleet, waits WaitIntervalInSeconds (at most 3,600 seconds) while watching the CloudWatch alarms in AutoRollbackConfiguration, then shifts the rest; any alarm returns traffic to the old fleet. LINEAR shifts in steps and ALL_AT_ONCE moves everything at once (with a baking period afterwards). Changing production variant weights by hand has no automatic rollback.
• AWS Lambda with AWS CodeDeploy: canary or linear deployment configurations (predefined, or custom with your own percentage and interval) shift alias traffic to the new version and roll back when a listed CloudWatch alarm fires.
• Rollback alarms should include a quality signal (for example a judge score computed on canary responses), not just errors and duration: a model update can be fast and error-free and still wrong.

Canaries need enough data

An alarm cannot fire on data points it never receives. A small canary released during a quiet period may receive only a few requests before the bake ends, and the alarms evaluate almost nothing. Fix the sample, not the alarm: make sure the new version receives enough representative requests during the bake, either by generating them (synthetic user workflows, covered below) or by baking long enough to cover normal traffic. Changes that make the alarm less sensitive, or that send even less traffic to the candidate, weaken detection instead of fixing it.

Shadow testing

• SageMaker AI shadow tests mirror each inference request to a shadow variant in real time, return only the production variant's response to the caller, show a live dashboard comparing both variants' latency and errors, and can capture shadow responses for offline quality comparison. You promote the shadow variant when satisfied.
• Bedrock models have no shadow variant. Mirror asynchronously: after returning the current model's answer, publish the request and answer to an Amazon SQS queue (or an event bus) whose consumer calls the candidate and stores both responses. The candidate's latency, throttling and errors never touch the user. Calling both models in parallel inside the request couples their failures and duration.

Deployment mechanics in general (endpoints, provisioned throughput, inference profiles) are covered in the lesson on model deployment strategies.

User-centred evaluation turns real users' and reviewers' judgements into data you can count, join to the exact response and feed back into datasets. Three mechanisms work together: in-app feedback, rating systems and annotation workflows.

Feedback interfaces tied to the exact answer

• Mint a unique ID for every answer, return it to the UI and store each rating against that ID. Users often rate an earlier turn minutes later, so session IDs and click timestamps cannot identify the rated answer.
• Carry the same keys into the logs. The Converse API's requestMetadata takes key-value pairs (for example the answer ID and prompt version) that are written to Amazon Bedrock model invocation logs and can be used to filter them. A rating can then be joined to the exact logged prompt and response, and thumbs-down rates grouped by prompt version, model or tenant.
• Ask why, with structure. After a thumbs-down, offer a short fixed list of reason codes (for example outdated information, wrong product, incomplete steps, unsafe suggestion). Reason codes can be counted and routed to the right fix; free-text comments and transcripts need someone to read them, and a star scale measures severity, not cause. An optional comment box can sit alongside the codes.

Annotation workflows: what is open to new accounts

For systematic review of sampled production answers (for example a monthly compliance review whose results become regression test cases), you need a workflow that shows each answer to qualified reviewers, records structured labels and combines several reviewers' labels into one trusted result. Amazon SageMaker Ground Truth and Amazon Augmented AI (A2I) are both no longer open to new customers; existing customers keep using them as normal, but AWS plans no new features. For a new design:

• Rating response quality: Amazon Bedrock human-based evaluation jobs (your own Cognito work team, custom metrics and rating methods, as above).
• Labeling or approving production answers at volume: a custom review UI, for example orchestrated by a Step Functions Standard workflow that pauses on a .waitForTaskToken task, storing labels in Amazon S3 or DynamoDB. Route each item to several reviewers and consolidate in your own code (majority vote, or adjudication of disagreements).

If your account already uses Ground Truth

An existing Ground Truth customer can keep running labeling jobs with a private workforce of its own reviewers. Pick the task type by what reviewers must mark:

Reliability comes from redundancy

Whichever tool you use, a single reviewer's mistake goes straight into your test set. Have more than one reviewer label each item and consolidate their labels into one result: in Ground Truth, set the number of workers per data object and use annotation consolidation (majority vote, or a custom post-annotation Lambda function); in a custom workflow, implement the same vote yourself. The share of items on which reviewers agree also measures how clear the rubric is. Any approach that ends with one reviewer's label per item gives up the redundancy.

A quality gate runs a fixed, versioned evaluation against every change to the model, prompt, retrieval settings or knowledge base and blocks the release when a score falls outside tolerance. The design details decide whether it catches real regressions without flagging noise.

Golden datasets must be versioned

A golden dataset is a curated set of inputs with reference answers or rubrics. Edit it in place and yesterday's score is no longer comparable with today's: a drop may only mean harder questions were added. Publish each revision under a new version (for example a new S3 key, or the S3 object version ID), record the dataset version with every result, and whenever the dataset changes, rebaseline by re-running the current production configuration on the new version. The set can keep growing. Nothing else restores comparability: a score can only be compared with a baseline produced on the same dataset version, so changing generation settings or tolerances does not substitute for a rebaseline.

Gate per slice, with noise-aware tolerances

• An overall score can rise while a small slice collapses (a product area, language or customer segment). Keep a baseline per slice and gate each one.
• Small slices vary between runs even with no change. Size each tolerance to the slice's sample size (wider bands for small slices) or use a statistical test, so a genuine regression in a small slice fails while normal run-to-run variation passes.
• The unit of the gate matters: individual questions are too noisy to gate on, and any pooled score lets a small slice's regression be diluted by the rest.

Wiring the gate into the pipeline

A typical pipeline (AWS CodePipeline or a Step Functions workflow) deploys the candidate configuration to a test stage, runs the evaluation (a Bedrock evaluation job or your own harness), compares each dimension and slice with its baseline, and fails the stage if any check fails. Per-dimension floors from the first section apply here too: gate factual accuracy, relevance and consistency separately rather than on an average.

Continuous evaluation scores a sample of the answers users actually received, on a schedule, with no daily human work, and publishes the result as a metric. It complements the golden-set gate: the gate tests fixed questions before release; continuous evaluation tests real traffic after it.

1. Source: Amazon Bedrock model invocation logging already writes each request and response to S3 (and/or CloudWatch Logs).
2. Schedule: Amazon EventBridge Scheduler starts a workflow (AWS Step Functions or Lambda) each day or night.
3. Sample: select a sample of yesterday's logged prompts with their responses and write them as an evaluation dataset.
4. Score: run a judge-based Bedrock evaluation job using those responses as your own inference responses, so the job skips inference and scores exactly what users saw. Re-invoking the model would score different answers.
5. Publish: write the scores as CloudWatch custom metrics, alarm on them, and keep the result files for reporting.

Stratify the sample

A simple random sample gives small segments too few items for a stable score: a segment carrying 2% of traffic gets about 10 items in a 500-item sample. To report each language, product or tenant, use stratified sampling: a fixed number per segment at the same total size, and compute any overall score by weighting each segment by its traffic share.

Operational monitoring of these metrics (dashboards, alarms, tracing, token and cost attribution) is covered in the lesson on monitoring GenAI applications.

Evaluation reporting turns job outputs into model-comparison and trend views that non-technical stakeholders can read without the AWS console, delivered automatically. The serverless pattern on AWS is Amazon Athena over the result files in S3, visualised in Amazon Quick Sight (the BI capability of Amazon Quick, formerly Amazon QuickSight).

• Query: Bedrock evaluation jobs write JSONL results to S3; Athena queries them in place (a Glue Data Catalog table over the prefix), with no cluster to run.
• Visualise: Quick Sight dashboards show model comparisons (bars per candidate and metric), trends over time and quality-cost scatters, with filters for business unit, use case or slice.
• Automate: scheduled email reports send a dashboard snapshot to readers on a cadence; readers need no console access. Email reports are an Enterprise edition feature.
• Restrict: row-level security attaches a rules dataset that maps users or groups to the field values they may see (for example business unit), so one dataset and one dashboard serve every audience and a new unit is a new rules row. Per-audience copies or views multiply assets; a filter control lets anyone pick any value.

Other tools have narrower roles: the Bedrock console report card covers one job; CloudWatch dashboards suit operational metrics for engineers but do not send scheduled email reports; Amazon OpenSearch Dashboards needs a domain to run.

Deployment validation proves that a new model version, prompt or configuration still works end to end and still behaves the same way where it should, before users depend on it. It combines synthetic user workflows with AI-specific output checks.

Synthetic user workflows

Amazon CloudWatch Synthetics canaries run scripted journeys (browser or API scripts) on a schedule: sign in with a test account, hold a multi-turn conversation, assert that the reply contains the expected content. They publish success metrics you can alarm on and run during deployments to generate traffic. CloudWatch RUM and AWS X-Ray observe real users' traffic and do not run scripts; Route 53 health checks only test reachability.

A canary against the public API during a traffic-shifting bake mostly hits the old version and already exposes users. To test the new version before anyone reaches it, use a CodeDeploy BeforeAllowTraffic hook for Lambda deployments: a validation function invokes the new version directly, runs the scripted conversation and reports Failed through PutLifecycleEventHookExecutionStatus, which fails the deployment before any traffic shifts. AfterAllowTraffic runs only after the shift completes.

AI-specific output validation

Replay a representative set of recent inputs through both versions. Each of these checks needs a measure of meaning or of support by the source; statistics about the surface of the text (shared words, length, filter counts) measure something else.

Separate real drift from sampling noise

At a non-zero temperature a model's answers vary from run to run, so comparing one new answer with one stored answer flags many prompts even when nothing changed. Test at the production settings: sample several answers per prompt from each version, compute the within-version similarity (normal variation) and the cross-version similarity, and flag a prompt only where cross-version similarity falls below the within-version level. The comparison is only meaningful at the settings users actually get, and the noise is handled by the within-version baseline rather than by a looser threshold.