AIP-C01 GenAI Monitoring: Bedrock Metrics, Invocation Logs, Tracing, Tool and Vector Store Observability

A complete GenAI observability system combines four layers that answer different questions: operational metrics (is the model call fast, available and within quota?), request-level logs (what exactly was sent and returned?), traces (which step of a multi-step agent or RAG pipeline did what, in what order?) and quality and business metrics (are the answers still good, and do they still produce the outcome the business pays for?). No single AWS feature covers all four, so Task 4.3 is about choosing the right source for each question and joining them.

The distinctions that decide most scenarios are about grain and timing. Metrics are aggregates and can never give you a single prompt back. Invocation logs hold content but say nothing about tool calls or the order of agent steps. CloudTrail records API calls and callers but never prompt or response content. Billing data is accurate and matches the invoice but arrives hours later and has no per-request detail. A monitoring design that needs "per request", "within minutes" and "matches the invoice" therefore needs more than one source.

This lesson covers monitoring. Reducing token cost belongs to the cost optimization lesson for Task 4.1, latency tuning to the performance lesson for Task 4.2, and turning on invocation logging as a compliance record to the governance lesson for Task 3.3; here those features appear only as signals to watch.

Amazon Bedrock publishes runtime metrics to the AWS/Bedrock CloudWatch namespace automatically, with the ModelId dimension: Invocations, InvocationLatency, InputTokenCount, OutputTokenCount, InvocationThrottles, InvocationClientErrors, InvocationServerErrors, TimeToFirstToken and EstimatedTPMQuotaUsage. Reading them correctly depends on knowing exactly what each one measures.

Latency: first token versus last token

• TimeToFirstToken is the time from the request to the first generated token. It is emitted only for streaming operations (ConverseStream and InvokeModelWithResponseStream). It is the metric for "nothing appears for a while after I press send" complaints, the delay a user perceives before streaming starts.
• InvocationLatency runs from the request to the last token. A longer answer raises it even when generation speed and the first token are unchanged.
• Output tokens per second (OTPS) is not a vended metric, but metric math gives it: OutputTokenCount / InvocationLatency (convert milliseconds to seconds). If latency rises while OTPS stays steady, the model is writing longer answers (often after a prompt change asking for more detail); if OTPS falls, generation itself got slower. This is the standard performance benchmark for separating the two causes.

Errors and throttles: what the counts leave out

Two documented counting rules make naive error-rate formulas wrong:

1. Throttled requests are not counted in Invocations, InvocationClientErrors or InvocationServerErrors. They appear only in InvocationThrottles. A formula of errors divided by invocations does not move during a throttling storm. An accurate failure rate adds InvocationThrottles to both the failures and the total attempts.
2. A request that fails before Bedrock resolves the target model is published without the ModelId dimension. A graph filtered to one model ID (or to an inference profile's ID) undercounts these early failures. Query the metric without the dimension to see the whole account's failures.

Two further edges of the same counting rules: a request rejected for exceeding a service quota is counted in InvocationClientErrors (it is not a throttle), and a request rejected during input validation is not counted in Invocations at all.

Fixing only one of the two still leaves the rate wrong, and changing the statistic (Sum versus Average) changes neither rule. EstimatedTPMQuotaUsage is an estimate of tokens-per-minute quota consumption. Throttling actually follows the upfront reservation of input tokens plus max_tokens for each request, which the estimate does not reflect, so treat it as a capacity hint rather than an exact headroom gauge, and never as a delay or error signal.

Model invocation logs are the only AWS source that holds the prompt and response of each Amazon Bedrock call, so any analysis that needs per-request content, such as "which prompts produced truncated answers?", starts there. When delivered to CloudWatch Logs, each record is JSON with fields including timestamp, requestId, operation, modelId, identity.arn (the calling IAM principal), input.inputBodyJson, input.inputTokenCount, output.outputBodyJson, output.outputTokenCount and any requestMetadata the caller attached. (Large bodies can be written to S3 instead of inline.)

CloudWatch Logs Insights queries these records directly. For example, to list requests that reached a 4,096-token maxTokens setting, with their prompts:

fields @timestamp, requestId, identity.arn, input.inputBodyJson, output.outputTokenCount
| filter output.outputTokenCount >= 4096
| sort @timestamp desc

The same records feed aggregations, such as summing input.inputTokenCount and output.outputTokenCount grouped by a metadata key. This is the grain rule from the first section in practice: anything aggregated or recorded without content cannot be turned back into individual prompts, and any capture switched on today covers only future requests.

Monitoring the logging itself

If invocation logs are your record of every prompt, a silent delivery failure is an incident. Bedrock publishes delivery metrics such as ModelInvocationLogsCloudWatchDeliveryFailure and ModelInvocationLogsCloudWatchDeliverySuccess (with S3 equivalents). An alarm when the failure metric's sum is above zero fires on any failed delivery whatever the cause: a logging role that lost permissions, a KMS key whose policy does not allow the role, or a deleted log group. The success metric falling to zero is ambiguous, because it also happens when there is simply no traffic. The general rule: alarm on the signal that is emitted at the moment of the failure, not on the absence of a success or on one of the possible causes.

Token usage is attributed in one of two places, and they serve different readers: invocation logs for engineering's per-request, near-real-time breakdowns, and billing (Cost Explorer, the Cost and Usage Report) for finance's numbers that match the invoice.

Practical consequences:

• A shared Lambda function with one IAM role serving many tenants cannot be split by identity.arn; every record shows the same role. Attach requestMetadata (for example workflow and customerId) on every call and group by requestMetadata.customerId in Logs Insights. Nothing has to be created when a tenant signs up.
• requestMetadata is a log field, not a resource tag: it cannot be activated as a cost allocation tag and never appears in Cost Explorer.
• Application inference profiles are resources that carry tags into billing once the tag key is activated as a user-defined cost allocation tag, which is how a business unit's Bedrock cost appears in Cost Explorer. They are a poor fit for daily-changing values such as tenants, because each value needs its own profile.
• Billing data has no per-request or per-end-user grain, whatever tag you use, and the vended AWS/Bedrock metrics cannot take custom dimensions such as a user ID.

A requirement that mixes the two readers (an invoice-matching chargeback and a near-real-time per-request breakdown) therefore needs both mechanisms at once; neither covers the other's grain.

Once usage is attributed, anomalies are detected in two places: token bursts at metric speed with CloudWatch, and unusual spend on billing data with AWS Cost Anomaly Detection. The two run on different data and different clocks.

Token bursts in CloudWatch

• Anomaly detection alarms. GenAI traffic usually has daily and weekly cycles, so a static threshold either fires at every peak or misses bursts in quiet hours. A CloudWatch alarm on InputTokenCount (or OutputTokenCount) that uses an anomaly detection band as its threshold learns the cycle and trend and alerts within a few metric periods, with no hand-tuned value. Watching tokens rather than Invocations catches bursts caused by a few very long prompts.
• Contributor Insights. To see who is driving a burst, a Contributor Insights rule over the invocation log group keyed on identity.arn and summing input.inputTokenCount gives a continuously updated top-N of callers, can be added to a dashboard and supports alarms on a contributor's share of the total. A metric filter that uses a high-cardinality field (a role ARN, a user) as a dimension instead creates a separate custom metric for every value, which you pay for.
• Not the right tools here: CloudWatch Logs anomaly detection looks for new or unusual log patterns, which long prompts do not create; CloudTrail Insights works on API call rates, not token volume; and the vended Bedrock metrics have no caller dimension.

Spend anomalies with AWS Cost Anomaly Detection

AWS Cost Anomaly Detection uses machine learning on your billing data to learn each spend pattern and alert on departures, reporting the root cause (account, Region, service and usage type). It suits workloads whose spend varies legitimately, where fixed AWS Budgets thresholds or a static CloudWatch billing alarm on EstimatedCharges keep firing on planned work. Trusted Advisor gives optimization advice, not anomaly detection.

• Monitors can track AWS services, linked accounts, cost categories or cost allocation tags. An AWS managed monitor on a dimension evaluates each value independently and picks up new values automatically (new accounts, new tag values). A customer managed monitor tracks a list you specify (for tag values, up to 10 in aggregate) and needs upkeep.
• Alert subscriptions are individual (as anomalies are detected) or daily/weekly summaries.
• Per-team detection in a shared account follows from the attribution section: tag each team's application inference profile, activate the tag key as a cost allocation tag, and create an AWS managed monitor on that cost allocation tag.

Because billing data lags by hours, Cost Anomaly Detection complements rather than replaces token-metric alarms when you need to know within minutes.

Quality KPIs (grounding or hallucination rate, judge scores, response drift, prompt effectiveness) are not vended by any service; you compute them and publish them as CloudWatch custom metrics, after which they behave like any other metric for dashboards, alarms, anomaly detection and metric math.

Publishing per-response signals without slowing the request

The CloudWatch embedded metric format (EMF) is a structured JSON log line that CloudWatch Logs turns into a metric asynchronously. A function that checks, say, whether each answer cites a retrieved source can write one EMF line with a WeakCitation value of 0 or 1 and a PromptVariant dimension. The metric's Average is then the weak-citation rate per prompt variant, which also makes it a prompt-effectiveness KPI.

Detecting gradual drift

Response drift is a slow change in output (longer, more hedged, less aligned with approved answers) with no deployment to blame. Two complementary signals catch it in production:

1. Cheap per-response measures, such as answer length and embedding similarity to approved reference answers, published as metrics and watched by anomaly detection alarms.
2. Periodic LLM-as-a-judge scoring of a sample of production answers (daily, for example), with the scores published as metrics and alarmed on drops.

What makes these drift monitors is that both are measured continuously on production traffic and compared with their own history. A control that acts on one response at a time, runs only at release time, or stores responses without measuring them cannot see a slow trend.

End-to-end GenAI tracing on AWS means emitting OpenTelemetry spans for each model call, tool call and agent step, and ingesting them into CloudWatch through CloudWatch Transaction Search, where the CloudWatch generative AI observability pages group them by agent and session.

Transaction Search: the one-time account setup

Before spans from Amazon Bedrock AgentCore or your own instrumented agents appear, the account must turn on Transaction Search once. It ingests 100% of spans as structured logs into the aws/spans log group, and indexes a configurable percentage of them as trace summaries for trace search (the default is a small sample). Two consequences matter:

• Every span is queryable with CloudWatch Logs Insights over aws/spans, even those that were not indexed. A question about all traces from last week (for example, every session in which a particular tool returned an error) is a Logs Insights query grouping spans by trace ID.
• Raising the indexing percentage affects only future spans; it does not re-index the past.

Model invocation logging and CloudTrail are separate records, and CloudWatch Application Signals does not replace Transaction Search for span ingestion.

Instrumenting the agent

To group traces by user session outside AgentCore Runtime, put session.id into OpenTelemetry baggage. X-Ray active tracing on a Lambda function records the function and its AWS SDK calls but not GenAI spans with model and tool semantics, so it does not populate the generative AI observability view on its own.

Tool calls made through an Amazon Bedrock AgentCore Gateway are monitored with the gateway's own CloudWatch metrics, logs and spans; multi-agent coordination is monitored by propagating one trace context and one session ID across every agent hop.

Gateway metrics: per tool, and split by where time is spent

The gateway publishes metrics in the AWS/Bedrock-AgentCore namespace (they are not shown on the CloudWatch generative AI observability page, so chart them yourself), including Invocations, Duration, Latency, TargetExecutionTime and error and throttle counts, with dimensions such as Name (the tool), Method (the MCP method, which aggregates all tool calls) and Resource (the gateway).

• TargetExecutionTime is the time spent executing the target (a Lambda function, an OpenAPI or Smithy target and so on); Duration covers the whole request from receipt to final response; Latency is the time to the first response token (the initial response time), not the full request. Compared per tool on the Name dimension, they show whether extra time is in the tool or in the gateway. A Lambda Duration metric cannot do this when one function serves several tools behind different targets.
• Usage baselines: an anomaly detection alarm on Invocations for each tool's Name learns that tool's daily cycle and growth, so a sudden surge in one tool's calls, whatever causes it, alerts within minutes. Alarming per tool matters because a spike in a rarely used tool disappears inside a gateway-wide total.

Seeing what each tool call actually carried

Gateway spans name the tool (tool.name) but do not store the arguments or response. For that, configure delivery of the gateway's APPLICATION_LOGS (to CloudWatch Logs, Amazon S3 or Firehose). Those records include the MCP request and response bodies and carry trace_id and span_id, so each record can be joined to the step in the agent's trace that made the call. Payloads exist only in these log records; metrics, spans and trace settings describe calls, not their contents.

Multi-agent coordination

When a supervisor agent invokes sub-agents on their own AgentCore runtimes with InvokeAgentRuntime, each hop starts a new, unrelated trace unless the supervisor passes context on. AgentCore Runtime accepts trace headers (W3C traceparent, and X-Amzn-Trace-Id) and the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id header. Forwarding the incoming trace context makes sub-agent spans children of the supervisor's trace, and reusing the user's session ID in the session header puts every agent in the same session view. Only these headers link the hops: an identifier placed anywhere else, such as in the request body or a span attribute, is just data to AgentCore.

Business-impact dashboards place an outcome metric that the application publishes next to the operational and token metrics, in one CloudWatch dashboard; forensic traceability comes from CloudTrail, configured to record the Bedrock APIs you will need to investigate.

Business impact next to operations

• Publish the outcome. Only the application knows that a business outcome happened, such as a booking confirmed or a refund approved without staff; it emits that as a custom metric (EMF keeps it off the request path).
• Compute ratios with metric math. With stable per-token prices, an expression such as (InputTokenCount × input price + OutputTokenCount × output price) / ConfirmedBookings at a 1-day period charts token cost per booking, and CloudWatch alarms accept metric math expressions. Metric math is the feature that does arithmetic across metrics; the query and ranking features group or order metrics but do not divide one by another, and billing metrics are neither timely nor per outcome.
• Several accounts, one view. CloudWatch cross-account observability links source accounts to a monitoring account, where one dashboard and its alarms use metrics from all of them. Batch exports into a BI tool such as Amazon Quick Sight (formerly QuickSight) suit trend reporting but are only as fresh as their last load.
• Compliance at a glance. Compliance monitoring belongs on the same dashboard: an alarm widget for invocation-log delivery failures, plus AWS Config rule compliance (for example, a custom rule that checks model invocation logging stays enabled) so a configuration drift is visible next to the operational picture. Setting up those controls is covered in the governance lesson for Task 3.3.

Forensic traceability with CloudTrail

CloudTrail records management events by default, and the model runtime calls InvokeModel, Converse and ConverseStream are management events, so a default trail already sees them. All Agents for Amazon Bedrock Runtime operations, including flow and knowledge base runtime calls, are data events, recorded only when you add advanced event selectors for their resource types:

Data events start recording when the selector is added; there is no backfill. A management-event selector cannot capture them, and invocation logging records model content, not CloudTrail coverage of these APIs. (Amazon Bedrock Agents is now positioned behind AgentCore for new agent builds, but existing agent aliases are audited this way.)

Counting streaming calls correctly

For a streaming call such as ConverseStream, CloudTrail writes the AwsApiCall event and, when the stream completes, a second event of type AwsServiceEvent with the same eventName, userIdentity, source IP and user agent, linked back through serviceEventDetails.parentRequestId and carrying completion details such as token usage. A count of calls per principal must filter on eventType = 'AwsApiCall', joining the service events by parentRequestId only when you need their details. The service event is not authorized or evaluated by IAM or SCPs, so it is never evidence of a call a policy failed to block. A trail that also records Bedrock data events receives a data-event copy of each streaming event as well, so filter on the event category too when counting.

Vector store operations for FM augmentation have three parts: performance monitoring (is retrieval fast and is capacity sufficient?), automated index optimization (keeping the index efficient as data changes) and data quality validation (did every document actually get ingested?).

Performance metrics

If SearchRequestLatency rises while SearchOCU sits at the configured maximum, the collection is capacity-bound and the limit should be raised. On a managed domain, alarm on the Maximum of KNNGraphMemoryUsagePercentage across data nodes at a level such as 80% to add capacity before the breaker trips. (Sizing that memory is covered in the vector store lesson for Task 1.4.)

Automated index optimization

• OpenSearch HNSW indexes with heavy delete-and-rewrite traffic accumulate many small segments containing deleted documents, so each query searches more graphs and latency climbs. A force merge scheduled after the write window (for example, EventBridge Scheduler invoking a Lambda function) reduces the segment count and purges deletes in place, behind the same index name. It rebuilds graphs for the merged segments, so it costs compute, but far less than re-ingesting; the k-NN warmup API afterwards loads the new graphs into memory before peak traffic. Search-time parameters and extra replicas change how the existing segments are searched, not how many there are.
• pgvector IVFFlat indexes on Aurora PostgreSQL learn their list centroids from the data present at build time. pgvector advises building IVFFlat after the table has data and sizing lists to the row count (about rows/1000 up to 1 million rows, then about sqrt(rows)). An index built on a small early table has unrepresentative centroids and too few lists as data grows, so recall falls and each probe scans a huge list. The routine is to build a replacement with CREATE INDEX CONCURRENTLY and a suitable lists value, then drop the old index, which does not block writes. The underlying rule: centroids and the list count are fixed when the index is built, so only a rebuild with new parameters changes them; query-time settings only trade latency for recall on the index you already have.

Knowledge base ingestion data quality

An Amazon Bedrock knowledge base ingestion job can finish as COMPLETE while individual files were skipped or failed. Two checks close that gap:

• Gate on the job statistics. GetIngestionJob returns IngestionJobStatistics (numberOfDocumentsScanned, numberOfNewDocumentsIndexed, numberOfModifiedDocumentsIndexed, numberOfDocumentsFailed, numberOfDocumentsDeleted and others). A workflow that must stop when any document failed checks numberOfDocumentsFailed > 0; the job status alone, or scanned counts that include failures, are not enough.
• See which files and why. Configure log delivery of the knowledge base's vended APPLICATION_LOGS (to CloudWatch Logs, S3 or Firehose). Resource-level records carry statuses such as RESOURCE_IGNORED, EMBEDDING_FAILED and INDEXING_FAILED with status_reasons, which a Logs Insights query lists per file and a metric filter can alarm on. Model invocation logs and CloudTrail data events describe query time, not ingestion.

GenAI applications fail in ways traditional ML systems do not: they hallucinate fluent but false answers, give different answers to the same question, and make reasoning errors partway through multi-step agent runs while every API call succeeds. Each failure mode has its own troubleshooting technique.

Golden datasets for hallucination regressions

A golden dataset is a fixed set of questions with verified reference answers. Run it on a schedule against the production configuration (daily, for example), not only as a deployment stage, because regressions can also come from documents that change in the knowledge base between deployments. Score each run automatically, for example with an Amazon Bedrock evaluation job using RAG metrics such as correctness and faithfulness, publish the scores as CloudWatch metrics and alarm on drops. Replaying yesterday's production prompts detects change but has no ground truth, and output token count is not a reliable hallucination signal. (Designing evaluation frameworks and deployment quality gates belongs to Task 5.1.)

Output diffing for consistency

Because wording varies between runs, consistency is measured in meaning, not characters:

• For a probe that sends each fixed question several times, embed the responses and publish the lowest pairwise cosine similarity per question as a custom metric. It stays high for paraphrases and drops when one run diverges. Embeddings can miss small contradictions (a negation, a changed number), so low-similarity or high-risk questions can be passed to an LLM-as-a-judge check as well.
• For a model-version change, run each prompt several times on both versions and compare cross-version similarity with each prompt's own within-version similarity; flag the prompt when answers across versions are less alike than answers within a version. A single fixed cutoff for every prompt confuses natural variation with real change, and a temperature-0 exact-text comparison flags paraphrases.

Reasoning path tracing

When an agent produces a wrong result, the question is which step introduced the error: the model's decision, the arguments it sent a tool, or the tool's result. With AgentCore observability and Transaction Search on, open the session's trace in the generative AI observability view and inspect the model and tool spans in order; that is the least-effort route for one session. For a pattern across all sessions in a past period, query aws/spans with Logs Insights (every span is there, unlike the sampled trace index). When arguments and responses are needed, join the gateway's APPLICATION_LOGS to the trace by trace_id and span_id. Invocation logs are not keyed to sessions and miss the tool side; metrics and CloudTrail carry no content.