AIP-C01 flashcards

Foundation Model Integration, Data Management, and Compliance

• Fix format with the prompt, put changing or citable facts in retrieval, fine-tune for a voice prompting cannot hold, and combine techniques when needs differ.
• Bedrock for managed serverless APIs; Custom Model Import for your own supported-architecture weights; a SageMaker AI endpoint for your own container, code or GPU type.
• Serverless inference has no GPU; asynchronous inference scales to zero but returns results later; batch inference and batch transform are for work nobody waits on.
• Priority is fastest without a reservation, Flex is cheapest per request, Reserved assures a baseline for one or three months with overflow to Standard.
• Take long or throttle-prone model calls out of the request: queue the work, store status, and push the result; use Step Functions Standard for jobs beyond Express or Lambda limits.
• Workflows for paths you can draw in advance, agents for paths decided at run time, and a workflow that calls an agent for one open-ended step.
• Measure PoCs with the right source: TimeToFirstToken for streaming, response usage and metrics per call, billed tokens for cost, and a KPI against a control group for value.
• Bedrock reserves input plus max_tokens against the quota at request start; quotas and model availability are per account and per Region.
• The Generative AI Lens is imported, published and shared as a custom lens; major versions wait for owners, minor versions apply automatically.

• Filter before you benchmark: modality, context window, maximum output, features, Region and lifecycle state rule models out; only then compare candidates on your own prompts.
• Context window limits what the model reads, maximum output limits what it writes, and quotas limit throughput: truncation, rejection and throttling point to different limits.
• Bedrock evaluation jobs compare candidates on your data: automatic metrics need reference answers, model-as-a-judge works without them, human jobs put your own raters in the loop.
• Legacy models take no new customers, no new Provisioned Throughput and no new fine-tuning; idle access can lapse; after EOL calls fail and nothing migrates you automatically.
• Converse makes a model switch a model-ID change, but features such as tool use and system prompts are per model, and model-specific fields go only in additionalModelRequestFields for that model.
• AppConfig gives fast, validated, gradual model changes with alarm rollback; the Lambda extension's poll interval sets how quickly warm functions see them.
• An API Gateway plus Lambda adapter, with routing in AppConfig and keys in Secrets Manager, lets providers change without client changes.
• Geographic profiles keep data in a geography; global profiles widen capacity and lower price. Allow the model in every listed Region, allow destination Regions (or unspecified for global) in SCPs, and remember profiles take no Provisioned Throughput.
• Retry transient faults, break the circuit on sustained ones with shared DynamoDB state and a half-open trial, and degrade to a smaller model or non-generative answers rather than failing.
• LoRA adapters share one base model as adapter inference components; Custom Model Import serves Hugging Face safetensors on demand; Model Registry approval drives the pipeline, and guardrails (canary, linear, rolling) roll back on alarms.

• Match each DQDL rule to its tolerance: IsComplete and IsUnique for 'always', ratios or 'with threshold > x' (the share that must pass) for 'mostly'.
• Row-level outcomes quarantine individual records; failing the job stops the whole run; EventBridge score events and CloudWatch rule metrics only describe the run, and the events need Publish metrics to CloudWatch selected.
• Compare drifting data with its own history: Metric > avg(last(k)) * factor for an explicit formula, DetectAnomalies for a learned check; both history features and analyzers run only in Glue ETL.
• Validate single requests or files in Lambda and publish dimensioned CloudWatch metrics; a handled validation failure is not a Lambda error.
• Transcribe: channel identification for separate channels, diarization for one mixed channel, custom vocabulary to fix recognition, EventBridge job events instead of polling.
• Bedrock Data Automation: standard output for generic extraction, blueprints for your own fields; SageMaker Processing with ShardedByS3Key for hours-long container preprocessing.
• Converse is model-agnostic (inferenceConfig plus additionalModelRequestFields); InvokeModel needs each provider's native body; batch inference needs .jsonl records of recordId plus modelInput, joined on recordId.
• SageMaker endpoints accept the container's own schema; large or slow requests go to asynchronous inference, which can scale to zero and notify through SNS.
• Re-send ordered, role-tagged history every turn and bound it with recent turns plus a running summary.
• Use code when the result must be exact, an FM when the task is language, and Comprehend (built-in or custom recognizer behind an endpoint) to add structured entities.

• pgvector in Aurora PostgreSQL when similarity must join relational metadata; DynamoDB pairs with a vector store via zero-ETL; S3 Vectors for huge, rarely queried sets, with hot indexes exported to OpenSearch Serverless.
• OpenSearch embeds for you with a Bedrock connector, a text_embedding ingest pipeline and neural queries; the connector role must trust OpenSearch, and its creator needs iam:PassRole (plus ml_full_access with fine-grained access control).
• A metadata file is named <file>.<ext>.metadata.json and sits beside the document; S3 object metadata and tags are not ingested as attributes.
• Range filters need NUMBER attributes (epoch dates); multi-tag domains need STRING_LIST with listContains; includeForEmbedding steers ranking while a filter excludes.
• Store-specific filter setup: JSONB custom metadata column with a GIN index on Aurora, keyword fields on self-managed OpenSearch, faiss on OpenSearch Serverless, non-filterable AMAZON_BEDROCK_TEXT/METADATA keys at S3 Vectors index creation.
• Primary shards (about 10–30 GiB each) set per-query work and are fixed at creation, so reindex behind an alias; replicas on more nodes add throughput.
• k-NN graphs are off-heap: size for every copy, use on_disk mode with rescoring under memory pressure, warm up new indexes, and bulk-load with refresh off and zero replicas.
• Systems without a connector use a Custom data source with Ingest/DeleteKnowledgeBaseDocuments; new Confluence, SharePoint, Salesforce and Web Crawler connectors need a managed knowledge base.
• Syncs are incremental, metadata-only changes skip re-embedding, and direct fixes on an S3 data source must also be written to S3.
• dataDeletionPolicy acts when a data source or knowledge base is deleted; deletion protection acts during a managed-connector sync.

• Diagnose first: a passage that is never retrieved needs better chunking, embeddings, query handling or hybrid search; a passage retrieved but ranked low needs reranking.
• Fixed-size chunking sets max tokens and overlap percentage; semantic chunking splits where meaning changes; hierarchical matches small children and returns their parents, so it can return fewer results than requested; no chunking keeps one file as one chunk.
• Chunking configuration is fixed per data source and the embedding model is fixed per knowledge base: changing either means a new data source or a new knowledge base, synced before cut-over.
• Own chunking rules = no chunking plus a custom transformation Lambda function; chunk-level metadata on native chunks = native strategy plus a custom transformation Lambda function.
• Embed documents and queries with the same model, dimension and settings; use Cohere search_document for passages and search_query for questions; cross-language search needs a multilingual model.
• Titan Text Embeddings V2 offers 1,024, 512 or 256 dimensions and float32 or binary vectors; binary plus a reranker gives the largest storage cut with ordering protected.
• OpenSearch dimension, engine, space type, m and ef_construction are set when the index is built; ef_search tunes recall per query with no reindex.
• Aurora as a knowledge base store needs pgvector, the RDS Data API, a Secrets Manager secret, an HNSW index, a GIN text index and, when used, a GIN index on custom_metadata.
• A reranker needs numberOfResults well above numberOfRerankedResults, and the Rerank API reranks candidates from any source.
• Search with the expanded or rewritten query but generate from the user's original question; managed QUERY_DECOMPOSITION exists only in RetrieveAndGenerate.

• Define roles and instructions once as a Prompt Management prompt; per-call differences go in {{variables}}, not copies.
• With a prompt version ARN as the Converse modelId, system, inferenceConfig, toolConfig and additionalModelRequestFields must live in the prompt; messages can be appended.
• Instructions guide; guardrails enforce. Use an Organizations Bedrock policy with a versioned guardrail when member accounts must not remove it, and ApplyGuardrail for models outside Bedrock.
• Structured outputs guarantee JSON shape; ranges, lengths and cross-field rules still need a Lambda check, and max_tokens can still truncate.
• Clarification that waits for a person needs a Standard workflow with .waitForTaskToken; Express workflows reject it and stop at five minutes.
• Comprehend multi-label for several intents per message, multi-class for one; a confidence threshold with a clarifying fallback needs no retraining.
• DynamoDB history: session partition key, timestamp sort key, TTL on each item, and a rolling summary to bound tokens.
• Prompt versions are the release unit; deny authors bedrock:CreatePromptVersion to make versioning the approval gate.
• CloudTrail data events record who rendered a prompt or ran a flow without content; only model invocation logging holds prompt and response text.
• In Bedrock Flows the first true condition wins, DoWhile defaults to 10 iterations, and rollback is repointing an alias to an earlier version.

Implementation and Integration

• Persistence and context are different jobs: an S3-backed session manager restores state across hosts; a conversation manager decides what the model sees.
• AgentCore Memory long-term records exist only with a strategy; put {actorId} in the namespace to keep each user's records separate on one resource.
• Summarize when early context may be condensed; pin the first message of a sliding window when it must stay word for word at no extra cost; bound tool results at the source.
• Workflow = fixed pipeline; graph = developer-defined paths chosen by data, with capped loops; swarm and agents-as-tools = model-chosen routes; Agent Squad = classifier routing over existing assistants.
• A ReAct loop continues on stopReason tool_use with a toolResult in a user message; cap iterations with a Choice state that routes to a final call without tools.
• Safeguards match symptoms: repetitive handoff detection for ping-pong, client-side timeouts for hung calls, a shared DynamoDB circuit breaker with one half-open trial for failing dependencies.
• Enforce boundaries in IAM, not prompts: session tags with dynamodb:LeadingKeys, permissions boundaries required by iam:PermissionsBoundary, AWS:SourceArn on action group functions.
• Pick models per subtask from evaluations and keep the mapping in AppConfig; aggregate labels by vote and free text by a rubric judge.
• Human review is a Standard workflow .waitForTaskToken callback with TimeoutSeconds and Catch escalation; A2I is closed to new customers.
• Validate business rules in the tool and return an error toolResult; filter MCP tools with an allow-list and resolve name clashes with prefixes.

• Irregular traffic on a Bedrock model: on-demand calls (for example from Lambda, with its 3-second default timeout raised); steady load needing reserved capacity: Provisioned Throughput.
• Provisioned Throughput bills hourly until deleted; no commitment can be deleted any time, 1-month and 6-month terms are cheaper per hour but cannot be deleted early.
• Send requests to Provisioned Throughput by putting its provisioned model ARN in modelId; agents use it through routingConfiguration on the alias; model units need a Support quota grant first.
• A custom-model Provisioned Throughput can be re-pointed to another custom model from the same base model, keeping the ARN and commitment.
• Bedrock Marketplace runs models on SageMaker AI endpoints but calls them through Bedrock APIs and tools; your own registered endpoint needs network isolation and unchanged provider artifacts.
• Serverless Inference has no GPUs; real-time endpoints scale to zero only with inference components, MinInstanceCount 0 and a step policy on NoCapacityInvocationFailures; async endpoints use HasBacklogWithoutCapacity.
• GPU memory is weights (2 bytes per parameter in 16-bit) plus a KV cache that grows with context length times concurrent sequences.
• Shard across an instance's GPUs with tensor parallelism; raise throughput with continuous batching; scale on high-resolution concurrent-request metrics, not CPU.
• Quantization may reduce accuracy; speculative decoding, compilation, fast model loading and continuous batching do not.
• Use the smallest model that passes evaluation per task; cascade with checks on the actual answer, or use a Bedrock prompt router for two same-family models with English prompts.

• EventBridge connections hold API credentials in Secrets Manager and reach private APIs through VPC Lattice resource configurations; API destinations time out after 5 seconds, Step Functions HTTP Tasks after 60 seconds.
• Use .waitForTaskToken in a Standard workflow when a legacy system finishes hours later and can call back; Express workflows cannot wait for callbacks.
• A custom EventBridge bus with a rule per consumer decouples producers; archive and replay reprocess history; filter events with patterns before they invoke a model.
• SQS FIFO message group ID = entity ID gives per-entity order with parallelism; FIFO deduplication spans only 5 minutes, so webhook idempotency needs a conditional write on the delivery ID.
• Webhook handlers acknowledge fast and queue the model call, and verify HMAC signatures over the raw body.
• Cognito user pools for app users behind a backend, identity pools with rules-based role mapping for direct SDK calls, IAM Identity Center with SCIM for workforce access across accounts.
• Serverless Bedrock models are enabled on first invocation; block unapproved models with an SCP deny using NotResource.
• ConverseStream needs bedrock:InvokeModelWithResponseStream; cross-Region inference profiles need the model allowed in every Region of the profile.
• Short-term Bedrock API keys for production; deny bedrock:CallWithBearerToken for LONG_TERM tokens to shut off long-term keys.
• Bedrock does not run on Outposts and VPC endpoints cannot live in Wavelength Zones; a gateway enforces token budgets, per-team metrics, requestMetadata audit tags and per-team application inference profiles.

• A long Bedrock call that fails at about a minute usually hit the SDK read timeout, so raise it on the client. In Java, apiCallAttemptTimeout caps one attempt and apiCallTimeout caps the whole call with retries.
• Container code uses the ECS task role, EKS workloads use Pod Identity on their service account, and on-premises servers with certificates use IAM Roles Anywhere.
• For SQS → Lambda → Bedrock, cap the event source mapping's maximum concurrency, set visibility to at least six times the function timeout, and return batchItemFailures.
• Use StartAsyncInvoke for one long generation written to S3, batch inference for many offline S3 records at a lower price, and on-demand calls for live traffic.
• Request validation, SDK generation, usage plans, response streaming and VTL AWS integrations are REST API features. HTTP APIs have none of them.
• ConverseStream sends text in contentBlockDelta, stopReason in messageStop and token usage in metadata. Tool input arrives in fragments, and errors can arrive after HTTP 200.
• Streaming to a browser needs an unbuffered path: a RESPONSE_STREAM function URL, REST STREAM mode (proxy only, no cache, compression or VTL) or a WebSocket API when the client must talk back.
• Retry throttling and 5xx errors with jittered backoff in one layer only, never validation or access errors. Adaptive mode limits the whole client instance.
• Per-client rate limits are usage plans with API keys, with method overrides for expensive model routes. Stage throttling is shared by every caller.
• Route with a static Converse-based table when the task names the model, Step Functions plus a Choice state when content decides, and CloudWatch throttles, server errors and latency when interchangeable models vary.

• REST API STREAM mode works on proxy integrations, lasts up to 15 minutes and removes caching, content encoding and VTL response templates, so those move into the backend.
• A Regional or private REST API can raise its 29-second integration timeout by quota request. HTTP APIs stop at 30 seconds and WebSocket routes at 29, and neither limit can be raised.
• When the client can neither wait nor read a stream, return 202 with a job ID and let it poll a status method. On WebSockets, acknowledge fast and push chunks to @connections.
• CountTokens is free and model-accurate. Output is billed per generated token, so raising maxTokens costs nothing for short answers. A request validator with maxLength rejects oversized input before the function runs.
• Retry throttling and model timeouts with capped backoff and jitter inside the deadline. Never retry a ValidationException, and raise the SDK read_timeout for long generations.
• Use a.conversation with AIConversation for chat with history, and a.generation with useAIGeneration for one typed result. a.ai.dataTool lists only the signed-in user's records.
• OpenAPI import with API Gateway extensions gives REST APIs integrations, request validators and SDKs (HTTP APIs have neither). put-rest-api merges by default, so use overwrite, then deploy.
• Bedrock Flows releases go draft to version to alias. Roll back by repointing the alias, and use an iterator before and a collector after to process arrays.
• Make CRM writers idempotent with Powertools. Cap a downstream API's concurrency with SQS and event source mapping maximum concurrency, and cache rotated secrets with the Parameters and Secrets extension.
• Distributed Map with MaxConcurrency and a tolerated failure rate handles huge S3 batches. Input/Output S3Uri gets around the 256 KiB state limit, and BDA blueprints with splitting extract per-document fields.

AI Safety, Security, and Governance

• Harmful or out-of-scope input with no custom code: a Bedrock guardrail. Denied topics for subjects in any wording, word filters for exact strings, content filters for harm.
• Content filter strength is set separately for prompts and responses; LOW still blocks high-confidence harm.
• Non-English filtering and prompt leakage detection need the Standard tier, which requires cross-Region inference.
• API Gateway request validators and AWS WAF are REST API features; WAF sees only up to its body inspection limit, so pair the limit with oversize handling Match.
• Human review that takes hours: Step Functions Standard + .waitForTaskToken. Sub-second verdicts the API waits on: synchronous Express via StartSyncExecution. A2I is closed to new customers.
• ApplyGuardrail (source INPUT or OUTPUT) applies guardrail policies to any model or content without invoking a model.
• Streaming that must never leak or must mask: synchronous guardrail mode.
• InvokeModel needs input tags with a fresh random tagSuffix for prompt attack filtering; Converse guardrails do not read tool results, so screen them with ApplyGuardrail.
• Contextual grounding needs grounding_source and query marked; grounding catches invented facts, relevance catches off-question answers. Automated Reasoning is detect-only and non-streaming.
• JSON Schema structured outputs guarantee shape, not truth; exact values come from the database (text-to-SQL, lookups) behind least-privilege access.

• Each Bedrock API family needs its own interface endpoint: bedrock-runtime for Converse/InvokeModel, bedrock-agent-runtime for Retrieve and InvokeAgent; enable private DNS so code is unchanged.
• Batch inference in a VPC = vpcConfig on the job (service role with ENI permissions) + S3 gateway endpoint + an aws:SourceVpce bucket policy.
• SageMaker AI network isolation cuts all container egress and credentials; extra files must ship inside the model artifact. Clients reach the endpoint through sagemaker.runtime.
• Private OpenSearch Serverless for a knowledge base: AllowFromPublic false, SourceVPCEs for your apps, SourceServices bedrock.amazonaws.com; who can do what lives in the data access policy plus IAM aoss:APIAccessAll.
• Callers you cannot change are covered by an enforced guardrail; the bedrock:GuardrailIdentifier deny only rejects callers that omit it and breaks RetrieveAndGenerate/InvokeAgent on the same role. Comprehensive content guarding ignores callers' tags.
• Tenant scope comes from verified identity and is enforced by a Retrieve metadata filter or by IAM (ABAC); Lake Formation data filters give column plus row security, but only once IAMAllowedPrincipals Super is removed.
• Who read an object or queried a knowledge base requires CloudTrail data events; KMS Decrypt disappears if the trail excludes AWS KMS events.
• Comprehend: DetectPiiEntities for offsets, ContainsPiiEntities for labels, async ONLY_REDACTION jobs for bulk copies. Macie finds PII in S3 but never fixes it: custom data identifiers plus EventBridge remediation.
• Guardrail ANONYMIZE keeps the content, BLOCK discards it; tool arguments, tool results and invocation logs are outside its reach, so use ApplyGuardrail and CloudWatch Logs data protection with logs:Unmask.
• Bedrock never trains on your prompts; zero retention needs data retention mode none per Region plus an SCP, and makes review-requiring models unavailable. Delete data with expiry mechanisms (Lifecycle with noncurrent expiration, log retention, TTL); use reversible tokens only when real values must come back.

• Exact prompt and response plus caller identity with no code change: Bedrock model invocation logging. It is off by default, per Region, and writes only to the same account and Region.
• With a CloudWatch Logs destination, bodies over 100 KB are kept only if an S3 large-data location is configured, and data protection masking does not reach those S3 objects.
• CloudTrail never holds prompt text. Converse and InvokeModel are management events; knowledge base retrievals, agents, flows and guardrail evaluations are data events that need advanced event selectors.
• Integrity of audit logs: CloudTrail log file validation. Immutability: S3 Object Lock in compliance mode.
• Org-wide guardrail with no code change: an Amazon Bedrock policy referencing a numeric guardrail version, shared with a resource-based policy, created per Region, without Automated Reasoning.
• bedrock:GuardrailIdentifier needs an explicit Deny with StringNotEquals on ARN:version, and must list InvokeModelWithResponseStream to cover streaming.
• Stop anyone but the platform role changing governance settings: an SCP with an aws:PrincipalArn exception. Restore within minutes: EventBridge on the CloudTrail event plus Lambda.
• Glue lineage needs Glue 5.0+ and Spark DataFrames; the SageMaker Unified Studio Import data lineage option is off by default.
• Knowledge Base attribution comes from .metadata.json sidecars and citations; metadata changes need a re-sync.
• Model card content edits create immutable versions; status changes do not; a Bedrock model's card is filled in entirely by you.

• Set enableTrace on InvokeAgent for reasoning traces; collaboratorName and callerChain attribute steps across collaborating agents.
• Traces contain prompts: keep them server side, persist them to encrypted, retained, access-controlled storage, and show users only vetted or mapped text.
• Faithful explanations come from recorded steps or validated schema fields, never from raw reasoning or a model describing itself.
• Cite from RetrieveAndGenerate citations or validated passage IDs; store original URLs as metadata attributes to link users to the source system.
• Confidence is measured (grounding, retrieval and self-consistency scores), published with EMF using low-cardinality dimensions, and calibrated per model version.
• Counterfactual tests vary one signal, use many signals per group, sample repeatedly and run both halves on the same versions.
• Fairness alarms use per-group metrics and metric math on the gap or ratio between groups, not an overall average.
• Pin A/B arms to prompt versions and serve flows through aliases; hash user IDs for sticky splits and tag metrics by variant.
• LLM-as-a-judge offers Stereotyping, Harmfulness and Refusal metrics, per-category scores, and bring-your-own responses with one model identifier per job.
• Guardrails block content; Lambda checks enforce required content and exact values; SageMaker Model Cards record your model's limitations and approval status.

Operational Efficiency and Optimization for GenAI Applications

• Match the lever to the driver: input tokens, output tokens, price per token, repeated invocations or idle capacity.
• Anything done after the model responds (trimming, discarding fields, dropping thinking blocks) saves nothing; billing happens at generation.
• Per-request tokens come from the response usage field; per-tenant spend needs tagged application inference profiles; per-request evidence needs invocation logs with requestMetadata.
• Bedrock reserves input + maxTokens against the quota up front, so an oversized maxTokens causes throttling; quota settles at input + cache writes + output × burndown.
• Control output with a length instruction plus a maxTokens cap, or a stop sequence at a known delimiter; temperature and topP do not change length.
• Bound chat history with recent turns plus a summary, return only needed tool fields, and retrieve wide then rerank to a few chunks.
• Compare models on cost per successful task from their own token counts on your data, including error-handling cost.
• Batch inference is discounted and low-overhead for bulk offline work, but cannot target provisioned models; Provisioned Throughput pays only when it stays busy.
• Prompt caching needs an identical, large-enough static prefix read again within its TTL; unique prefixes make it cost more.
• Response caches skip the model: exact keys must include every output-affecting input, semantic caches need context scoping and invalidation, and CloudFront cache keys must exclude cookies and tracking parameters.

• Latency ≈ time to first token + output tokens ÷ tokens per second: long input raises TTFT, long output raises total time.
• TimeToFirstToken is published only for ConverseStream and InvokeModelWithResponseStream calls.
• Predictable questions with inputs ready ahead of the peak are a pre-computation job: generate once, serve from a store.
• Latency-optimized inference is a per-request performanceConfig setting for supported models; over its quota it falls back to standard latency and standard rates.
• Parallelize independent FM calls, stream the output a person reads, and benchmark TTFT and output speed at high percentiles under real load.
• Fix the query before tuning the index: metadata filters, query decomposition, standalone rewrites and identifier normalization.
• Hybrid search needs a normalization processor; weights follow sub-query order and constant boosts vanish after normalization.
• Put restrictive filters inside the knn clause, and lower ef_search (no rebuild) when recall is surplus and latency is high.
• Bedrock reserves input + max_tokens at request start and settles at input + cache writes + output × burndown; oversized max_tokens throttles early.
• Batch what can wait, cap fan-out where work enters (Map MaxConcurrency, SQS maximum concurrency), reuse SDK clients and size their connection pools.

• TimeToFirstToken exists only for streaming calls; InvocationLatency runs to the last token; OutputTokenCount / InvocationLatency separates slower generation from longer answers.
• Throttled requests are excluded from Invocations and the error metrics, and pre-routing failures carry no ModelId: an accurate failure rate adds InvocationThrottles and queries without ModelId.
• Only model invocation logs hold per-request prompts, responses and token counts; alarm on ModelInvocationLogsCloudWatchDeliveryFailure above zero to catch silent logging gaps.
• requestMetadata gives per-request attribution in logs with no resources to create; tagged application inference profiles with activated cost allocation tags give invoice-matching cost in Cost Explorer.
• Use CloudWatch anomaly detection alarms and Contributor Insights for token bursts within minutes, and AWS Cost Anomaly Detection (AWS managed monitors pick up new values) for spend on billing data.
• Publish quality KPIs with embedded metric format (no extra API call); a 0/1 metric's Average is a rate usable in alarms and metric math.
• Transaction Search ingests 100% of spans into aws/spans but indexes only a sample, and indexing changes apply to future spans only; forward traceparent and the runtime session ID header across agents.
• AgentCore Gateway metrics by Name split per tool, with TargetExecutionTime vs Duration; gateway APPLICATION_LOGS hold tool payloads joinable by trace_id and span_id.
• An ingestion job can be COMPLETE with failed files: gate on numberOfDocumentsFailed and read RESOURCE_IGNORED/EMBEDDING_FAILED records in the knowledge base APPLICATION_LOGS.
• Run golden datasets on a schedule against production, and compare outputs by semantic similarity against each prompt's own baseline rather than exact text.

Testing, Validation, and Troubleshooting

• Score FM answers semantically (BERTScore, embedding similarity), not lexically, and gate each quality dimension separately so one cannot mask another.
• Bedrock automatic jobs: one task type per job; accuracy and robustness (score change under prompt perturbation) for every task type, toxicity for all but text classification; custom JSONL with prompt, referenceResponse and category (per-category results), up to 1,000 prompts.
• Human-based Bedrock jobs compare up to two sources (including your own inference responses); Likert comparison gives direction and strength of preference; API-created jobs need a flow definition with source AWS/Bedrock/Evaluation.
• Pick a model by filtering on hard thresholds, then optimising the objective; compare cost per successful outcome, not list price or cost per request.
• A/B tests assign per user (hash plus TargetVariant on SageMaker, hash plus an AppConfig flag for Bedrock); canaries need rollback alarms with a quality signal and enough traffic; shadow tests never expose users.
• Tie feedback to a per-answer ID passed in Converse requestMetadata and collect reason codes; SageMaker Ground Truth and A2I are closed to new customers, so new designs use Bedrock human evaluation jobs or a custom review workflow, with several reviewers per item and consolidated labels.
• Version golden datasets and rebaseline on change; gate per slice with tolerances sized to sample size; score logged production responses on a schedule with stratified samples.
• Validate updates with Synthetics canaries or a CodeDeploy BeforeAllowTraffic hook, faithfulness-based hallucination rates and embedding-based drift checks run at production temperature.

• Retrieve-only jobs score the retriever (context relevance, context coverage — coverage needs reference answers); retrieve-and-generate jobs score answers (faithfulness, correctness, completeness, citations, refusal). Run both on the same questions to tell a retrieval fault from a generation fault.
• Faithfulness is judged against the retrieved passages, so a true but unsupported detail lowers it while correctness stays high.
• Citation precision falls on wrong extra citations; citation coverage falls on unsupported or missing citations.
• External pipelines are evaluated with bring-your-own responses that include the retrieved passages; one RAG source per job, so compare configurations with separate jobs on the same questions, metrics and evaluator.
• Custom metrics need {{prediction}} in retrieve-and-generate jobs and {{context}} in retrieve-only jobs; referenceContexts is read only through {{reference_contexts}} in a custom metric.
• Trust a judge only after measuring agreement with expert ratings; swap pair order for position bias and give explicit criteria for length bias.
• Expert ratings come from Bedrock human-based evaluation jobs with your own work team (ComparisonLikertScale captures preference strength); A2I is closed to new customers.
• Evaluate rankers at the k the model receives (recall at that k, MRR), verify provenance by source location and metadata, measure Retrieve latency alone at p95 under load, and re-tune score thresholds after changing embeddings.
• AgentCore Evaluations: GoalSuccessRate (session, optional assertions) for task completion; ToolSelectionAccuracy and ToolParameterAccuracy need no ground truth; trajectory exact / in-order / any-order are deterministic and need an expected trajectory. Classic Bedrock Agents are judged from their InvokeAgent trace.
• Online evaluation samples live traffic into CloudWatch with no ground truth; on-demand checks specific sessions; batch compares past time windows; a custom evaluator used by an enabled online configuration is changed by cloning it.

• Isolate the failing layer before changing anything: content handling, API integration, prompt, retrieval or the production template.
• An input-too-long ValidationException is fixed by budgeting input tokens (measured with CountTokens, which is free and model-specific), not by lowering maxTokens or cutting strings.
• stopReason max_tokens means raise the output cap or continue; model_context_window_exceeded means shrink the input.
• Long documents need structure-aware splitting measured in the model's tokens, because token density differs by language; retrieval cannot guarantee full coverage.
• AccessDeniedException in a fresh account can mean missing Marketplace subscribe permissions or the Anthropic use case form; intermittent denials through a cross-Region profile mean the model is not allowed in every destination Region.
• Converse keeps one request shape across models; InvokeModel bodies are provider-specific; every toolUse needs a matching toolResult.
• Invocation logs keep bodies inline only up to 100 KB (configure S3 large-data delivery), correlate through requestMetadata, are per Region, and do not capture bedrock-mantle calls.
• Roll back a prompt by pointing at the previous Prompt Management version ARN, then compare versions on fixed inputs with identical model settings.
• Call the Retrieve API to separate retrieval from generation; a custom generation template without $search_results$ sends no passages.
• Silent relevance loss usually means mismatched embeddings (input type or model), drift, flattened tables, misnamed metadata sidecars, pgvector post-filtering or an operator that does not match the index.

Yes — the whole AIP-C01 deck is free and needs no account, as is every AIP-C01 revision lesson. Create a free account to save progress and drill practice questions; full-length mock exams are a one-time unlock per certification.