Evaluating ML and GenAI Models: Metrics, Drift, Explainability and RAG on AWS (MLA-C02)

An experiment is reproducible when you can rebuild its result from what you recorded: the code revision, the exact data, the parameters, the environment and the metrics it produced. On AWS that record is kept in managed MLflow on SageMaker AI for model training, and in Amazon Bedrock Prompt Management (plus repeatable Amazon Bedrock evaluations) for prompts and foundation models. Ad hoc records can hold numbers, but only an experiment tracker gives you run comparison, lineage and a model registry in one place.

Managed MLflow on SageMaker AI

SageMaker AI hosts open-source-compatible MLflow for you (a serverless MLflow App or a tracking server), so there is nothing to patch or scale. Each run belongs to an experiment and records parameters (mlflow.log_param), metrics (mlflow.log_metric), artifacts such as plots and the model itself, and tags. mlflow.autolog() captures hyperparameters, metrics and the trained model for common libraries automatically. The MLflow UI then compares runs side by side, and the MLflow Model Registry holds the chosen model as a registered, versioned model (how registered versions are approved and released is Task 3.3).

• Connecting from a training job. A notebook in SageMaker AI Studio may already be wired to MLflow, but a training job is a fresh container. The script must call mlflow.set_tracking_uri() with the tracking server's (or App's) ARN, and the job needs the sagemaker-mlflow plugin (for example in requirements.txt) so MLflow can authenticate with the execution role's IAM permissions. If either is missing, MLflow silently falls back to local file storage inside the container, which is discarded with the container, so the job still succeeds while the runs are lost. Keep the channels apart: training-job metric definitions feed CloudWatch, and MLflow receives only what the script logs to the tracking server.
• Logging what you need to rebuild a model. Parameters and metrics describe a run but cannot recreate it. Log the training data as an MLflow dataset input (mlflow.log_input), which records its source location (for example an S3 URI) and a content digest so you can tell whether the data changed, and tag each run with the Git commit ID of the code it executed. Data version + code version + parameters + environment is what makes a run reproducible.

Amazon Bedrock Prompt Management

Prompt Management stores a prompt (template text, variables, model and inference settings) as a managed resource with two kinds of state:

A version cannot be edited. To fix a released prompt, correct the draft, create a new version from it and repoint the application at the new version's ARN; the old version stays intact as the rollback target. Deleting a version breaks any caller that uses its ARN. For scored, repeatable comparisons of prompts or models across a whole dataset, run an Amazon Bedrock evaluation job (covered below) on a fixed prompt dataset, so the only thing that changes between runs is the thing you are testing.

A performance baseline is the reference a production model is compared with: statistics and constraints computed from the training data, and quality metrics (accuracy, recall, RMSE and so on) measured on held-out data before release. Drift detection then compares live traffic and live outcomes with that baseline on a schedule and alerts when they move apart. Build the baseline from the data the model learned from, not from recent production traffic, which may already have drifted.

Delayed ground truth is the hard case. If fraud labels arrive weeks later, a new fraud pattern that looks like normal spending leaves every input-distribution metric flat while recall falls. Only model quality monitoring catches it: log each prediction with an inference ID, join the late labels by that ID, compute recall (or another quality metric) and alarm when it drops below the baseline. The principle: a monitor can only detect a change in what it measures, so a change confined to the input-to-label relationship needs a monitor on outcomes. Retraining is a response to detected drift, never a substitute for detecting it.

Which tools to use

Amazon SageMaker Model Monitor is no longer open to new customers (existing customers keep using it; no new features are planned). For an account that already has it, the workflow is: enable data capture on the endpoint (requests and responses written to Amazon S3), run a baselining job on the training dataset to produce statistics.json and suggested constraints.json, then create a monitoring schedule. Its monitor types are data quality, model quality (which merges ground-truth labels with captured predictions), bias drift and feature attribution drift; violations surface in reports and CloudWatch metrics.

For a new account, AWS's documented replacement is built from open building blocks: endpoint data capture, a scheduled job (EventBridge with Lambda, or a SageMaker AI Pipeline step) that runs Evidently drift and quality presets against the training baseline, results logged to a SageMaker AI MLflow App next to the training metrics, Amazon SNS notifications when a drift or quality check breaches its threshold, optional QuickSight dashboards, and Amazon CloudWatch for endpoint metrics (latency, errors, utilization) and anomaly-detection alarms on custom metrics. Keep the two layers distinct: infrastructure metrics say whether the endpoint is healthy, while data and quality checks say whether the model is still right.

A shadow variant receives a copy of live requests sent to a SageMaker AI endpoint, but its responses are never returned to callers; a production variant serves real responses to the share of traffic its weight assigns. Use a shadow test when a new model must be judged on real traffic with zero customer impact; use weighted production variants (A/B testing, covered with production monitoring in Task 4.1) when you are ready for some customers to receive the new model's answers.

Running a shadow test in SageMaker AI:

• Endpoint type matters. Shadow tests need an instance-based real-time endpoint. They are not supported on serverless inference, asynchronous inference, multi-model or multi-container endpoints, AWS Marketplace containers or Inf1 instances. A model hosted on an unsupported endpoint type has to be on an instance-based real-time endpoint before a shadow test can run.
• Compare per request with data capture. Aggregate metrics describe each variant's latency and errors; comparing what the two models actually predicted needs per-request records. Enabling data capture for the test writes requests and both variants' responses to S3 for a request-by-request comparison. SageMaker AI copies the traffic to the shadow variant itself, so callers change nothing.
• Promote the winner. When you complete the test you can choose to deploy the shadow variant, which replaces the production variant on the same endpoint, so clients keep calling the same endpoint name with no changes.

What a shadow test adds over any offline evaluation is measurement under real, concurrent production load: live latency, error rates and the actual request mix.

Explaining a model means attributing its outputs to its inputs, either for one prediction (local explanation) or across a whole dataset (global explanation). The standard tool is SHAP (Shapley additive explanations): for each prediction it assigns every feature a signed contribution relative to a baseline, and the contributions add up to the difference between the prediction and the baseline prediction.

Taking the mean of absolute SHAP values matters: positive and negative contributions would otherwise cancel. Only local attributions answer a per-person question; any global ranking, however it is computed, describes the model as a whole. A PDP sets one feature to each value in turn for every row and averages the predictions, so it answers "what happens if we change X" questions such as a price change. Explanation techniques describe the model's behaviour; statistics computed on the data alone describe the data, not what the model learned.

SageMaker Clarify is no longer open to new customers (existing customers keep it). New accounts compute the same things with the open-source shap library in a notebook or processing job, bias formulas in pandas or scikit-learn, and fmeval or Amazon Bedrock evaluations for foundation models. The concepts and metric names are the same either way.

Post-training bias metrics for predictive models

Post-training bias metrics compare the trained model's predictions across groups (facets). They differ in what they condition on, which is why equal overall approval rates can hide unequal treatment:

• DPPL (difference in positive proportions in predicted labels): the gap in predicted approval rates between groups, regardless of true outcome.
• RD (recall difference): the gap in true positive rate, that is, among people whose true outcome was positive (for example, who repaid), how often each group was approved.
• DRR (difference in rejection rates): among predicted rejections, how often the rejection was correct (true negatives) in each group.
• DPL (difference in proportions of labels) is a pre-training metric on the training labels, not on model predictions; pre-training bias is part of data preparation (Task 1.3).

A model fails to converge when its training loss does not settle at a low value: it diverges to NaN, oscillates, or stays flat. Read the loss curve and the gradients first, then change the one thing the symptom points to. (Overfitting, where training loss is fine but validation loss rises, is a generalization problem covered with training in Task 2.2.)

The principle: match the fix to the mechanism. Regularization targets overfitting, clipping caps gradients that are too large, and activation and initialization choices address gradients that shrink toward zero. A change aimed at a different mechanism leaves the symptom in place.

Tools for watching and inspecting training

SageMaker Debugger is no longer open to new customers (existing customers keep its built-in rules, such as loss-not-decreasing and vanishing-gradient). For a new account:

• Alert: publish validation loss as a training job metric with a metric definition (a regular expression that SageMaker AI applies to the job's logs), then create an Amazon CloudWatch alarm on it.
• Inspect: write TensorBoard summaries (scalars plus per-layer weight and gradient histograms) to the job's output in S3 and open them in TensorBoard on SageMaker AI.
• Compare runs: log the loss curves to MLflow so a diverging run can be compared with a healthy one and its parameters.

Choose signals that measure learning (loss, gradient and weight statistics). Resource metrics and endpoint monitoring answer other questions: whether hardware is busy, or whether a deployed model is drifting.

Choose the metric that matches the cost of each kind of error in the business problem; a single headline number such as accuracy can rank models wrongly. For classification, start from the confusion matrix (true and false positives and negatives); for regression, decide how much large errors should count.

Two habits decide most metric choices. First, under class imbalance, report minority-class recall and precision (or PR AUC) instead of accuracy or specificity. Second, a threshold metric such as F1 at 0.5 is only meaningful once the operating threshold is known; until then compare models with a threshold-free metric. For regression, a model can have a lower RMSE but a higher MAE than another, when a few huge errors are pulling RMSE up. If each unit of error costs the same and the spikes are handled elsewhere, the lower MAE wins. The gap between RMSE and MAE describes how errors are distributed, not whether a model over- or underfits.

These are reference-based NLP metrics: each scores generated text against one or more human-written reference texts, but they measure different things. BLEU and ROUGE count shared word sequences (n-grams); BERTScore and embedding similarity compare meaning through embeddings, so they credit correct paraphrases.

Choose by what "good" looks like. If correct outputs share wording with the reference, as in translation, BLEU is standard; for summaries, ROUGE. If correct outputs are routinely paraphrased ("cancel your subscription" versus "end your plan"), n-gram scores fall even when editors approve, and you move to BERTScore or embedding similarity. Embedding similarity with a fixed embedding model is also repeatable and inexpensive, unlike a judge model, whose scores depend on another model's judgment. Classification F1 is a different metric from the token-overlap F1 used for short question answering.

Pick the Amazon Bedrock evaluation job by who or what does the scoring: a fixed algorithm (automatic), a judge model (LLM-as-a-judge, and RAG evaluations), or people (human-based). Each job scores model or RAG responses on a prompt dataset you supply, or on built-in datasets.

Automatic jobs can use built-in datasets, such as RealToxicityPrompts for toxicity, or your own prompt dataset with reference answers. A guardrail is not an evaluation: it filters content at runtime and produces no per-prompt scores.

Evaluating models that are not on Bedrock. Bedrock evaluations can score responses you generated elsewhere. Run the prompts through your model (for example, a fine-tuned LLM on a SageMaker AI endpoint), put the outputs in the prompt dataset as your own inference responses, and the job scores them alongside or against a Bedrock model, without moving or re-hosting your model. The job itself generates responses only from models it can invoke in Bedrock, so bringing your own responses is how any other model is included.

Human evaluation: human-in-the-loop quality assessment

A human evaluation job sends each prompt and the responses of up to two models (or inference sources) to a work team you manage, so confidential prompts are seen only by your own people, such as attorneys or medical reviewers. You define the metrics, write instructions, and set the number of workers per prompt so that several reviewers rate each item and one reviewer's bias is diluted; all ratings land in one report. The rating method decides what kind of judgment you collect:

Use human review where judgment is genuinely expert or subjective, and use it to calibrate automated judges. A human evaluation job is rated only by its human work team; when you want both kinds of score, run a separate LLM-as-a-judge job.

LLM-as-a-judge evaluation uses a separate evaluator model to score each response against named criteria and explain the score, so subjective quality can be measured at scale. Select metrics by the failure you need to track, because each built-in metric targets one failure mode.

Custom metrics cover rules no built-in metric knows, such as approved product names or a required disclosure. You write the judge prompt that states the rules and a defined rating scale, and the job applies it to every response. Built-in metrics apply generic criteria; organization-specific rules reach the judge only through a custom metric.

Making a judge trustworthy

Judge models have systematic biases: self-preference (favouring outputs from their own model family), position bias (favouring the first of two answers) and verbosity bias (favouring longer answers). If the judge consistently disagrees with a blind expert review, the evaluation is not trustworthy. Use a judge from a different model family from the candidates, and calibrate it by scoring a sample with both the judge and human experts and checking agreement. A systematic bias is fixed by independence and calibration; a change that keeps the same judge family or only adds randomness leaves the bias in place.

Bias detection and content-quality validation

For generative output, bias is measured on the text: run an LLM-as-a-judge job with the Stereotyping metric on representative prompts to get a scored report per prompt, and use Harmfulness for harmful content more broadly. Evaluate on prompts representative of your own application: a generic benchmark describes the model in general, not your assistant's behaviour. Guardrails are the runtime control that blocks content; they complement evaluation but produce no evaluation report. For predictive models, bias is measured with the post-training metrics (DPPL, RD, DRR) covered in the explainability section.

RAG quality is evaluated in two layers, and the metrics tell you which layer is failing: retrieval (did the knowledge base return the right passages?) and generation (did the model use them faithfully?). Amazon Bedrock RAG evaluations provide a job type for each, for Amazon Bedrock Knowledge Bases or for your own RAG pipeline (by bringing your own retrieved passages and responses).

Localizing a fault. Before touching prompts, run a retrieve-only job: low context relevance or coverage points at chunking, embeddings, search type or the number of results (retrieval tuning is Task 2.2; knowledge base configuration and reranking are Domain 3). If context relevance and coverage are high but faithfulness is low, retrieval is fine and the generator is adding unsupported claims; fix generation, for example by instructing the model to answer only from the retrieved passages and to cite them, or by changing the model. The principle: fix the failing layer and leave the working one alone.

Citation metrics catch two different citation failures:

• Citation precision: of the citations given, how many actually support their statement. Low when citations point at the wrong passages.
• Citation coverage: how much of the response is supported by its citations. Low when claims are grounded in the retrieved text but carry no citation.
• Faithfulness asks whether claims are grounded in the retrieved context at all, cited or not.

In production, keep the same evaluation dataset and rerun these jobs after each change to the knowledge base, embedding model or prompt, and track the scores over time with the other experiment results so that a drop in retrieval accuracy is caught before users report wrong answers. Runtime observability of live GenAI traffic in CloudWatch belongs to monitoring (Task 4.1).