Testing, Validation, and Troubleshooting practice questions

An assessment framework for FM output scores several independent dimensions (relevance, factual accuracy, consistency, fluency). They must be reported and gated separately because they trade off against each other.

Release 12 improved fluency and lost factual accuracy, and the average let one cancel the other. A gate with a floor on each dimension blocks any release in which a critical dimension regresses, whatever happens to the others. A higher or reweighted average still allows compensation, and a single holistic score removes the diagnostic detail altogether.

1. See that one dimension fell while another rose.
2. Recognise that an average lets dimensions offset each other.
3. Gate each dimension on its own threshold.
4. Report the dimensions side by side so the cause is visible.

Exam tip: Gate GenAI quality per dimension; an average lets fluency hide factual errors.

Automatic (programmatic) Bedrock model evaluation jobs offer three metrics: accuracy, robustness and toxicity. Robustness measures how stable the output is under small semantic-preserving perturbations of the input.

The requirement is about stability under typos and formatting changes. Robustness perturbs each prompt (character and case changes, whitespace) and reports the change between the original and perturbed scores, for example deltaF1 for question answering. Accuracy only scores the original prompts, toxicity looks at harmful content, and coherence belongs to judge-based evaluations.

1. Identify what varies: the input's surface form, not its meaning.
2. Recall the three automatic-job metrics: accuracy, robustness, toxicity.
3. Match input perturbation to robustness.
4. Read the delta score as sensitivity to noisy input.

Exam tip: Typos and casing changes in input = robustness metric (perturbation delta).

A custom prompt dataset for an automatic Bedrock evaluation is a JSONL file with prompt, referenceResponse (for accuracy and robustness) and an optional category key.

Adding category to each record makes Bedrock compute and report metric scores per category alongside the overall score, so one job yields the per-kind view. Separate jobs work but break the single-job constraint. A job has exactly one task type, and S3 tags are not part of the dataset format.

1. Recall the custom dataset keys: prompt, referenceResponse, category.
2. Map 'per kind of document' to the category key.
3. Reject options that need several jobs or are not read by the job.
4. Rerun the single job with categorised records.

Exam tip: Want per-segment scores from one Bedrock evaluation job? Add category to each JSONL record.

Human-based model evaluation jobs in Amazon Bedrock compare up to two inference sources, rated by your own work team. Either source can be a Bedrock model or your own inference responses (bring your own inference responses).

The SageMaker model's summaries can be generated ahead of time and placed in the prompt dataset as precomputed responses with a matching modelIdentifier. The job then shows each note's two summaries to the clinicians in the AWS-provided portal with the rating methods you configure. A judge-based job removes the humans; importing the model is unnecessary work; Ground Truth needs a custom UI.

1. Note the constraints: human raters, side by side, no custom UI, no migration.
2. Recall that human evaluation jobs accept two sources.
3. Use bring-your-own inference responses for the non-Bedrock model.
4. Assign the clinicians as the job's work team.

Exam tip: Non-Bedrock model in a Bedrock human evaluation? Precompute its responses and bring them in the dataset.

Bedrock human evaluations offer rating methods: thumbs up/down, Likert scale - individual, Likert scale - comparison, choice buttons and ordinal rank. Each captures a different kind of judgment.

The director wants direction and strength of preference between two responses. Likert scale - comparison asks for exactly that on a 5-point scale. Choice buttons and ordinal rank give only the winner; individual Likert and thumbs ratings judge each response on its own instead of comparing them.

1. Identify the judgment needed: a preference between two, with strength.
2. List the rating methods and what each records.
3. Exclude methods that give only a winner or only absolute ratings.
4. Configure the comparison Likert scale.

Exam tip: Preference plus strength between two models = Likert scale - comparison.

Human-based Bedrock evaluation jobs run their rating tasks through a SageMaker AI human review workflow (flow definition). The console builds it for you; the API requires you to build it.

For API-created human jobs, AWS documents that you must create a flow definition with AwsManagedHumanLoopRequestSource set to AWS/Bedrock/Evaluation, your private work team ARN and the SageMaker-owned evaluation task UI ARN, and that the Bedrock service role must be able to access the flow definition's output bucket. The CORS rule is the console path's requirement, which makes it the tempting distractor. Ground Truth jobs and invocation logging are separate features.

1. Separate what the console does for you from what the API needs.
2. Recall the flow definition with the AWS/Bedrock/Evaluation request source.
3. Check the service role's access to the flow definition output bucket.
4. Discard console-only and unrelated features.

Exam tip: API-created Bedrock human evaluation = your own flow definition + service role access to its output bucket.