SaveMyCert
Log in
5 of 5 free questions left today·for 30 a day
Databricks GenAI · Domain 6

Evaluation and Monitoring practice questions

Evaluation and Monitoring is worth 12% of the Databricks GenAI exam — the 5th-heaviest of the 6 domains. Evaluating and monitoring: quantitative model selection, agent evaluation with MLflow scoring, tracing and custom scorers, judges and ground truth, SME feedback, inference tables, Agent Monitoring, AI Gateway and cost control. Official weighting 12%. 6 fully worked examples are further down this page, answers included.

Exam weight
12%
the 5th-heaviest of the 6 domains
Questions
72
across 2 topics
Free, no account
5/day
a free account allows 30 a day
Explanations
Every option
right and wrong

Build a practice session

5 free questions left today.

Domains

How many?

Mode

Ready when you are

10 fresh questions drawn across 1 of 6 domains, in Learn mode.

Focused review

Every question you answer incorrectly, and every question you flag while practising, is saved here automatically. Finish a session and you can come back to re-drill just those.

6 sample Evaluation and Monitoring questions, fully explained

Questions from the Databricks GenAI bank mapped to domain 6, with the answer key and the reasoning behind every option. None of them repeat the examples on the main Databricks GenAI practice page.

Question 1Evaluation and Monitoring

A Generative AI Engineer is planning MLflow evaluation for a RAG agent whose retriever step is traced. Subject-matter experts have budget to label only some records with expected facts. The engineer wants to know which built-in judges depend on that labeling before they can report anything useful. Which TWO judges require ground truth in the records' expectations? (Select TWO.)

Choose TWO.

  • a
    RetrievalGroundedness

    RetrievalGroundedness compares the response with the retrieved context in the trace and needs no labels.

  • b
    Correctness Correct

    Correctness checks the response against expected facts or an expected response, which come from expectations.

  • c
    RetrievalRelevance

    RetrievalRelevance judges each retrieved chunk against the request and needs no labels.

  • d
    RetrievalSufficiency Correct

    RetrievalSufficiency checks whether the retrieved context contains the facts needed for the expected answer, so it needs expectations.

  • e
    RelevanceToQuery

    RelevanceToQuery compares the response with the request only.

The concept

Ground-truth judges compare against expectations; the others judge inputs, outputs and retrieved context alone.

Why that’s the answer

Correctness and RetrievalSufficiency both need to know what the right answer contains: one compares the response with it, the other checks whether the retrieved context covers it. RetrievalGroundedness and RetrievalRelevance sound similar but only relate the response or chunks to the request and the retrieved context, which the trace already holds. RelevanceToQuery uses the request and response only.

How to reason it out
  1. For each judge, ask what it compares the response or context with.
  2. If it compares with what the answer should contain, it needs expectations.
  3. Correctness and RetrievalSufficiency qualify; the others use trace data only.

Exam tip: Needs to know the right answer = needs expectations: Correctness, RetrievalSufficiency.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

Question 2Evaluation and Monitoring

A Generative AI Engineer evaluates an HR assistant on 600 questions across leave, payroll and benefits. Experts wrote, for each question, the rules a good answer must follow, and those rules differ by question: a leave answer must name the approval form, while a payroll answer must state the cut-off day. The engineer wants each response judged only against its own question's rules. Which scorer setup fits?

Choose one.

  • a
    Guidelines, with the experts' rules for the 600 questions combined into one judge's list

    A single Guidelines judge applies all its rules to every response, so leave answers would fail payroll rules.

  • b
    Correctness, with each record's rules stored as expected facts in its expectations

    Correctness checks whether facts appear in the answer; rules about form and behaviour are not facts to match.

  • c
    ExpectationsGuidelines, with each record's rules stored as guidelines in its expectations Correct

    This judge reads per-record guidelines from expectations, so each answer is held only to its own rules.

  • d
    Three Guidelines judges, one per category, each holding the combined rules of that category's questions

    Category-level judges still apply every rule in the category to each answer, and rules differ per question.

The concept

Guidelines applies the same criteria to every row; ExpectationsGuidelines applies criteria stored per row.

Why that’s the answer

When the criteria vary per record, they belong in the record: ExpectationsGuidelines reads the guidelines from each record's expectations and judges that response only against them. A global Guidelines judge, or one per category, applies every rule it holds to every response it sees, which produces false failures when a rule does not apply. Correctness is built for matching expected facts, not for checking behavioural rules.

How to reason it out
  1. Notice that the rules differ question by question.
  2. Global Guidelines would apply all rules to every answer.
  3. Per-record criteria live in expectations, which ExpectationsGuidelines reads.

Exam tip: Same rules for all rows = Guidelines; different rules per row = ExpectationsGuidelines.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

Question 3Evaluation and Monitoring

A Generative AI Engineer evaluates a retail chatbot on 300 unlabeled questions. Legal requires that no response names a competitor or promises a delivery date, and these rules apply to every answer. The team wants an LLM judge to check them without writing any labels. Which scorer should the engineer use?

Choose one.

  • a
    Guidelines, given the two rules in natural language Correct

    Guidelines judges every response against rules passed to the judge and needs no per-record labels.

  • b
    ExpectationsGuidelines, reading rules from each record

    This judge reads guidelines from each record's expectations, which would mean labeling all 300 records.

  • c
    Correctness, given the rules as expected facts

    Correctness needs expected facts per record and checks factual agreement, not compliance with rules.

  • d
    Safety, configured with the two legal rules

    Safety checks for harmful content; it is not the judge that takes custom natural-language rules.

The concept

The Guidelines judge applies natural-language rules that hold for every response.

Why that’s the answer

Rules that apply to every answer are passed once to a Guidelines judge, which checks each response against them using only inputs and outputs. ExpectationsGuidelines would need the rules copied into every record. Correctness compares with expected facts, and Safety is a fixed harmful-content check.

How to reason it out
  1. The rules are the same for every answer.
  2. No labels are to be written.
  3. Global natural-language rules go to the Guidelines judge.

Exam tip: Rules for every answer, no labels: Guidelines judge.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

Question 4Evaluation and Monitoring

A Generative AI Engineer prepares evaluation data for mlflow.genai.evaluate. Each record looks like this:{"inputs": {"question": "How do I reset MFA?", "region": "EU"}}The engineer will pass the agent through predict_fn so MLflow calls it once per record. How should the predict_fn be defined?

Choose one.

  • a
    def predict_fn(inputs: dict) -> str

    MLflow unpacks the inputs dictionary into keyword arguments; it does not pass it as one argument named inputs.

  • b
    def predict_fn(question: str) -> str

    MLflow passes every inputs key as a keyword argument, so the unexpected region argument makes the call fail.

  • c
    def predict_fn(request: dict, region: str) -> dict

    The parameter names must match the inputs keys; there is no request key in these records.

  • d
    def predict_fn(question: str, region: str) -> str Correct

    MLflow passes the inputs dictionary to predict_fn as keyword arguments, so its parameters match the keys.

The concept

mlflow.genai.evaluate calls predict_fn with each record's inputs unpacked as keyword arguments.

Why that’s the answer

The inputs field must be a dictionary whose keys are the predict_fn parameter names, because MLflow calls predict_fn(**inputs). A function taking one inputs dict, one that omits a key such as region, or one with differently named parameters fails or receives the wrong values.

How to reason it out
  1. Look at the keys inside inputs: question and region.
  2. Recall that predict_fn receives them as keyword arguments.
  3. Pick the signature whose parameter names match those keys.

Exam tip: inputs keys = predict_fn parameter names.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

Question 5Evaluation and Monitoring

A Generative AI Engineer received a Delta table of 2,000 questions and the answers a vendor's hosted model produced last month. The vendor contract has ended, so the model can no longer be called. The team wants to score those stored answers with Correctness and Safety in MLflow. How should the engineer run the evaluation?

Choose one.

  • a
    Pass records with inputs and expectations to mlflow.genai.evaluate without a predict_fn

    Without outputs, a trace column or a predict_fn there is no response for the scorers to judge.

  • b
    Pass records with inputs, outputs and expectations, plus a predict_fn that returns each stored output

    The evaluate contract says predict_fn must not be provided when outputs are present; a predict_fn would replace the stored outputs instead of scoring them.

  • c
    Serve the stored answers from a Model Serving endpoint and evaluate that endpoint

    Building an endpoint to replay a table is unnecessary; the stored outputs can be scored as they are.

  • d
    Pass records with inputs, outputs and expectations to mlflow.genai.evaluate without a predict_fn Correct

    When records already contain outputs, predict_fn must not be provided and the scorers judge the stored answers.

The concept

mlflow.genai.evaluate can score pre-computed outputs: supply an outputs column and no predict_fn.

Why that’s the answer

The data argument accepts records with inputs, optional outputs and optional expectations. If outputs are present, predict_fn must not be provided and the scorers judge those outputs directly. Leaving outputs out gives the scorers nothing to judge, adding a predict_fn alongside outputs breaks the evaluate contract and would replace the stored answers, and serving the table is far more effort than the task needs.

How to reason it out
  1. The model cannot be called, so only the stored answers can be scored.
  2. Records can carry outputs alongside inputs and expectations.
  3. With outputs present, omit predict_fn.

Exam tip: Answers already exist? Put them in outputs and leave predict_fn out.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

Question 6Evaluation and Monitoring

A Generative AI Engineer pulled 150 production requests with mlflow.search_traces into a DataFrame that has a trace column. The goal is to see whether agent version 7 answers those same requests better than the version that produced the traces. The mlflow.genai.evaluate documentation says predict_fn must not be provided when the data has a trace column. How should the engineer set up the comparison?

Choose one.

  • a
    Keep the trace column and remove predict_fn so the scorers run on the stored traces

    This respects the contract but scores the old version's answers, not version 7's.

  • b
    Keep the trace column and predict_fn, and add an outputs column with empty values

    An outputs column also forbids predict_fn, so this still violates the contract.

  • c
    Build records from each trace's request as inputs, drop the trace column, and keep predict_fn Correct

    Version 7 must generate new answers, so the records need inputs and no trace column, because a trace column forbids predict_fn.

  • d
    Merge the traces into a Unity Catalog evaluation dataset, keep predict_fn, and pass the trace column

    Passing the trace column alongside predict_fn is what the contract forbids, wherever the data comes from.

The concept

A trace column means 'score these recorded runs' and forbids predict_fn; re-running a new version needs inputs only.

Why that’s the answer

mlflow.genai.evaluate treats a trace column as finished executions to score, and its contract says predict_fn must not be provided with it. Removing predict_fn satisfies the contract but evaluates the old version's answers. To compare version 7 on the same requests, the records must carry the original requests as inputs, without traces or outputs, so predict_fn can run version 7 on each one.

How to reason it out
  1. Identify the goal: new answers from version 7 on old requests.
  2. Recall that a trace or outputs column forbids predict_fn.
  3. Keep the requests as inputs, drop the trace column, keep predict_fn.

Exam tip: Score what happened: pass traces. Score a new version: pass inputs plus predict_fn.

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges — the lesson that teaches this.

What Databricks GenAI domain 6 tests, topic by topic

The official exam guide breaks Evaluation and Monitoring into 2 topics. The question bank follows the same split, so a weak topic shows up as a cluster of misses you can go back and read.

Published Databricks GenAI practice questions per topic in Evaluation and Monitoring
TopicWhat it coversQuestions
Evaluating LLMs and agentsOfficial exam guide (March 18, 2026), Section 6 objectives: Select an LLM choice (size and architecture) based on a set of quantitative evaluation metrics; Evaluate agent performance using MLflow scoring and tracing; Identify evaluation judges that require ground truth; Use Databricks custom Scorers for evaluating agents and LLMs; Incorporate SME feedback to improve agent performance.36
Monitoring, inference logging and cost controlOfficial exam guide (March 18, 2026), Section 6 objectives: Select key metrics to monitor for a specific LLM deployment scenario; Use inference logging to assess deployed RAG application performance; Use Databricks features to control LLM costs; Use inference tables and Agent Monitoring to track a live LLM endpoint; Use AI Gateway (Inference Tables, Usage Tables, and rate limiting) to track an LLM or agent deployed via Agent Framework.36
Total72

Revise Evaluation and Monitoring before you drill it

Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and JudgesExpect scenario questions in which a Generative AI Engineer has a concrete requirement (no judge-model cost, deterministic results, a model that can no longer be called, non-coding reviewers, a memory budget, thresholds followed by an objective) and must pick a model, a scorer, a data setup or an SME workflow. Model-selection items give a small table of evaluation results with architectures, latency or cost and reward applying every threshold before the objective. Harness items show record dictionaries, predict_fn signatures, endpoint URIs or short scorer code and test what MLflow passes and returns. Judge items contrast near-twins such as RetrievalSufficiency and RetrievalGroundedness, Guidelines and ExpectationsGuidelines, expectations and feedback, and code scorers versus trace-aware LLM judges. SME items cover labeling schemas, dataset sync behaviour, Review App chat access and the conditions for judge alignment. Some items ask you to select two answers.Databricks GenAI Engineer: Monitoring, Inference Tables and LLM Cost ControlExpect scenario questions that describe a deployed LLM, RAG application or agent, a stakeholder's need and a constraint, then ask which metric, log, query, setting or cost lever fits. Logging items turn on what each record holds and when: payloads versus token counts versus DBUs versus traces, delivery delay, null oversized payloads, unlogged error responses, and how fallbacks and traffic splits appear through served_entity_id. Attribution items ask which join or field identifies an endpoint, a caller or a tenant. Monitoring items ask what an agent deployment sets up, how scorers are started, backfilled or written so they run in production, and which judge the logged data can support. Rate-limit items test scope, precedence and where token limits can be set; cost items test reading a usage pattern and choosing the lever that fixes it. Several options are often real features; requirements such as no stored text, no code change, no new identities, keep one endpoint, no self-built job or monitor without blocking decide between them.

Other Databricks GenAI domains

Domain 1: Design Applications14% of the exam · 84 questionsDomain 2: Data Preparation14% of the exam · 84 questionsDomain 3: Application Development30% of the exam · 179 questionsDomain 4: Assembling and Deploying Applications22% of the exam · 132 questionsAll Databricks GenAI practice questionsMixed sessions across all 6 domains — the way the real exam interleaves them.

Evaluation and Monitoring: your questions

Evaluation and Monitoring is domain 6 of the Databricks GenAI exam guide and carries 12% of the scored content — the 5th-heaviest of the 6 domains. On a 45-question paper that works out to roughly 5 questions, though Databricks does not publish an exact per-domain count and individual exam forms vary.

Source

The domain weight and topic list on this page come from the official Databricks GenAI exam guide.