Monitoring ML Models, FMs and Agents in Production: Drift, A/B Tests and Bedrock Evaluations (MLA-C02)

Pick the tool from the signal that is failing. Task 4.1 covers four different kinds of problem, and each has its own AWS mechanism: operational health (is the model answering, how fast, with what errors), data and model drift (has the input or the model's behaviour moved since approval), live experiments (is version B better than version A on real users) and generative AI quality (are the answers, retrievals and agent decisions any good).

Building dashboards, X-Ray and AgentCore Observability troubleshooting and token-cost tracking belong to Task 4.2; securing endpoints and Amazon Bedrock Guardrails belong to Task 4.3. This lesson covers detection and measurement.

Amazon Bedrock publishes runtime metrics to the AWS/Bedrock CloudWatch namespace for every model call, with no setup, and CloudWatch generative AI observability turns them into prebuilt views. The Model Invocations view shows invocation counts, token usage by model, latency percentiles, throttles and errors out of the box. Its per-request table (open one call, read its prompt and response) is filled only from model invocation logs delivered to CloudWatch Logs; logging only to Amazon S3 leaves that table empty. The Bedrock AgentCore view shows agent sessions, traces and error rates.

Choosing the right metric

The ModelId dimension undercounts early failures. Bedrock publishes runtime metrics both with and without a ModelId dimension. A request that fails before Bedrock resolves the target model (for example a malformed model or inference profile identifier) is counted only in the series without ModelId, so a per-model alarm can stay quiet during an outage. Alarm on the dimensionless metric too.

Agents hosted outside AgentCore Runtime

The AgentCore view reads OpenTelemetry spans and logs. To show an agent running on Amazon ECS, Amazon EKS, Lambda or elsewhere in the same place, enable CloudWatch Transaction Search in the account (the prerequisite for AgentCore observability) and instrument the agent with the AWS Distro for OpenTelemetry (ADOT) SDK, exporting spans and logs to CloudWatch.

Model invocation logging records the full request and response of every Bedrock call, and it is off by default. It is configured once per account and Region, with CloudWatch Logs, Amazon S3 or both as destinations; you choose which data types to include (text, image, embedding, video). CloudTrail is not a substitute: it records who called which API, never the prompt or response bodies.

• Large bodies: a log event holds a JSON body up to 100 KB. Larger bodies are delivered only if you configure an S3 location for large data delivery; the log event then references the S3 object. The symptom of a missing large-data location is log events with metadata and token counts but no body for the biggest requests.
• Attribution: each record carries identity.arn (the calling principal, which is identical for every application behind one shared role and session name) and requestMetadata, caller-supplied key-value tags you pass on Converse or InvokeModel calls. Adding an application tag to requestMetadata is a small code change that makes records filterable in Logs Insights with no new resources. Application inference profiles are the resource-based way to separate traffic and cost per application: each one is an AWS resource you create, and its ARN appears as the modelId.
• Delivery health: if a role or log group breaks, calls still succeed but logs stop arriving. Alarm on the delivery-failure metric for each destination you use: ModelInvocationLogsCloudWatchDeliveryFailure, ModelInvocationLogsS3DeliveryFailure and, when a large-data location is configured, ModelInvocationLargeDataS3DeliveryFailure.

CloudWatch Logs Insights queries over the log group let you filter by model, metadata, latency or token count and read the exact prompts behind a quality complaint.

SageMaker Model Monitor detects drift by comparing captured production traffic with a baseline on a schedule. Every monitor type needs data capture enabled on the endpoint (or batch transform job), which writes requests and responses to Amazon S3, plus a baseline and a monitoring schedule. Results include violation reports and, optionally, CloudWatch metrics for alarms. Model Monitor works only on SageMaker AI endpoints and batch transform; it cannot be attached to an Amazon Bedrock model.

Data quality violations

The data quality baseline job computes statistics and suggests constraints. Each run reports violations by check type:

• data_type_check: a feature's inferred type differs from the baseline.
• completeness_check: the share of non-null values fell below the baseline.
• missing_column_check: a column in the baseline is absent.
• baseline_drift_check: the distribution distance from the baseline exceeds the threshold. A shift in a feature's values that keeps its type, its non-null share and its column (for example an upstream change in how a value is scaled) trips only this check.

Bias drift versus feature attribution drift

Bias drift watches outcomes across groups: a post-training bias metric, such as the difference in positive prediction rates between two values of a facet, moving away from its baseline. Feature attribution drift watches how the model decides: Clarify computes SHAP attributions on live data and Model Monitor compares the feature-importance ranking with the baseline using an NDCG score. A model can keep accuracy and input distributions stable while either one moves. A Clarify pre-training bias job only describes the static training set and cannot detect production changes.

Model quality monitoring measures real-world accuracy by merging captured predictions with ground-truth labels that you upload later, which is the only way to track precision or recall when outcomes arrive days or weeks after the prediction.

1. Tag each request: pass a unique InferenceId on InvokeEndpoint. It is stored in the captured record, and your application keeps it with the business record (for example the transaction).
2. Upload labels: when the outcome is known, write ground-truth records keyed by the same inference ID to the S3 location the model quality schedule reads.
3. Merge and compare: the monitor joins captured predictions to labels by ID, computes metrics for the problem type and compares them with the baseline constraints, emitting CloudWatch metrics you can alarm on.

The baseline is different from a data quality baseline. Passing training features and labels to the data quality baselining job produces feature statistics only, never precision or F1 thresholds. A model quality baseline needs a dataset containing the model's predictions (and probabilities for threshold metrics) next to true labels, typically from running the current model on a held-out validation set, and a model quality baselining job told the problem type (BinaryClassification, MulticlassClassification or Regression) and which columns hold the inference and the ground truth. Unlabelled captured traffic cannot produce these metrics, and hand-written thresholds are not a measured baseline.

InferenceId only labels a request for this join; it does not route traffic.

Detection becomes useful when it triggers action without polling code. Event-driven wiring through CloudWatch and Amazon EventBridge covers both drift-triggered retraining and failure alerts.

Drift-triggered retraining

Configure the monitoring schedule to publish its metrics to CloudWatch, create an alarm on the drift metric for the features that matter, and create an EventBridge rule matching the alarm's state change to ALARM with a SageMaker pipeline execution as its target. The alarm state is the trigger because it changes only when drift crosses the threshold; the job finishing says nothing about whether drift was found.

Failures in pipelines and jobs

SageMaker Pipelines emits execution and step status change events, and training, processing and transform jobs emit job state changes, to EventBridge. A rule on a Failed status with an Amazon SNS target notifies on-call within minutes. AWS Config evaluates resource configuration, not run outcomes.

Anomalies that are not failures

Use production variants when users should receive the new model's answers in a controlled share; use a shadow variant when no user may see them. Both run on one endpoint and report CloudWatch metrics per variant.

Controls you are expected to know

• Weights: set initial weights per variant in the endpoint configuration (90/10 sends 10 percent of requests to the second variant).
• UpdateEndpointWeightsAndCapacities changes weights (and instance counts) on the running endpoint in place with no new endpoint configuration. UpdateEndpoint also avoids downtime but requires a new endpoint configuration.
• TargetVariant on InvokeEndpoint pins one request to a named variant, overriding the split for that request only (useful for QA). Setting it on every request replaces the endpoint's random split with client routing. CustomAttributes are passed to the container and do not route; InferenceComponentName targets inference components, not variants.
• InvokedProductionVariant in the InvokeEndpoint response names the variant that served the request. CloudWatch has per-variant operational metrics only; any outcome measured by the application itself has to be joined to the variant that produced it, and this response field is what makes that join possible.

Shadow tests

A SageMaker AI shadow test copies a configurable percentage of live requests to the shadow variant (useful when the candidate runs on costly instances), returns only the production variant's responses, shows a console dashboard comparing both variants' latency and errors, and lets you promote the shadow variant to production at the end. Because it uses copies of real requests, a shadow test reflects the live input mix as it is today.

For foundation models on Amazon Bedrock, detect input drift from the invocation logs, because Model Monitor data capture does not exist for Bedrock. Prompts are free text, so compare them in embedding space: a scheduled job embeds the recent logged prompts, compares their distribution (centroid distance, cluster proportions or a similar statistic) with launch-period embeddings, and publishes the result as a custom CloudWatch metric that can be alarmed on.

• Token counts only measure prompt length; new topics can have the same length profile.
• Re-running an evaluation on a fixed dataset of old prompts measures the model, not whether incoming questions changed.
• Embedding drift detects that the traffic moved; evaluation (next section) tells you whether answer quality suffered as a result.

Amazon Bedrock evaluations measure foundation model and RAG quality, and the job type follows from who judges and what is being judged.

Bring your own inference responses

Judge-based and RAG jobs accept a dataset that already contains each prompt's response, so Bedrock skips invocation and scores the answers as they were delivered. This is how you score logged production conversations, or a model hosted outside Bedrock (a SageMaker AI endpoint, a LangChain RAG stack), without generating new answers. Anything that invokes a model again scores fresh answers, not the ones users actually received.

Matching metrics to defects

To compare chunking strategies, isolate retrieval with a retrieve-only job; retrieve-and-generate metrics mix in the generator's behaviour. Human jobs use a private work team managed through Amazon Cognito; Amazon Augmented AI (A2I) is a production review loop for individual predictions, not a model comparison.

Agents fail in ways that request metrics do not see: a tool returns an error the agent politely explains, a supervisor routes to the wrong collaborator, or a stream ends early with HTTP 200. Each needs a specific signal.

Tracing what the agent did

Enable trace on InvokeAgent for an Amazon Bedrock agent. Trace events cover pre-processing, orchestration (the model's rationale, the action group invocation input with its parameters, and the observation it returned) and failure traces. With multi-agent collaboration, the supervisor's orchestration trace records each hand-off as structured fields: an invocation input of type AGENT_COLLABORATOR (collaborator name, alias ARN, input text) and an observation with the collaborator's output; trace parts also carry collaboratorName and callerChain. CloudTrail shows only that calls happened, Lambda logs miss the agent's side, and invocation logs hold the same story as raw prompt text.

Handled tool failures

Lambda's Errors metric counts only invocations that end in an unhandled error. A tool function that catches an exception and returns a message reports success, and the agent's own invocation succeeds too, so neither Lambda nor AWS/Bedrock/Agents error metrics move. To alert without changing what users are told, emit a custom metric for each handled failure (for example with CloudWatch embedded metric format log lines) and alarm on it.

Truncated streaming responses

With ConverseStream, the HTTP status is sent before tokens flow, so 200 never proves the answer finished. A complete stream ends with a messageStop event (carrying stopReason) followed by a metadata event with usage and latency.

Automated agent quality: AgentCore Evaluations

Online evaluation continuously scores a sampled share of live sessions (a sampling rate you set; the default is 10 percent) from an AgentCore-observed agent's traces, using LLM-as-a-judge evaluators. On-demand evaluation scores the sessions you hand it when you call it. Built-in evaluators work at different levels, and the level must match the problem:

• Session: Builtin.GoalSuccessRate (did the conversation achieve the user's goal).
• Trace: Builtin.Helpfulness and similar per-reply qualities.
• Tool: Builtin.ToolSelectionAccuracy and Builtin.ToolParameterAccuracy (right tool, right arguments) for silent delegation errors that raise no exception.

You can add custom evaluators. By default results are written as JSON events to /aws/bedrock-agentcore/evaluations/results/<config-id> (with session and trace IDs, so Logs Insights can list the lowest-scoring sessions) and scores are published as metrics in the Bedrock-AgentCore/Evaluations namespace, so a standard CloudWatch alarm pages on a falling score with no code.