Databricks GenAI Engineer: Evaluating LLMs and Agents with MLflow Scorers and Judges

You choose an LLM by measuring every candidate on the same representative evaluation set with the same scorers, recording operational metrics such as latency and cost alongside the quality scores, and then applying the application's requirements in order: drop every model that misses a hard threshold, and only then let the stated objective (lowest cost, highest quality, highest throughput) pick among the survivors.

The evidence has to come from this application. Public leaderboards, general benchmarks and a model card's parameter count describe general capability, not how a model answers your users' questions against your quality bar. Building a different evaluation dataset for each candidate breaks the comparison too: the dataset, the scorers and the judge configuration must be identical so the only variable is the model.

Thresholds first, objective second

1. List the hard requirements (for example a minimum Correctness mean, a minimum RetrievalGroundedness mean, a latency ceiling such as p95 under a set number of milliseconds, a memory budget).
2. Eliminate every candidate that fails any one of them, even by a hundredth. A model that is cheap, fast and nearly good enough is still out.
3. Rank the survivors on the objective the stakeholders named. If only one model survives, the objective never gets a say.

The common mistakes are treating a gate as the objective (choosing the most grounded model when groundedness was only a floor and correctness was the goal), and ignoring a gate because a model excels on the objective.

Which offline metric fits which task (for example ROUGE for summarisation) is taught in the evaluation-metrics lesson of the application development domain; here the focus is using the numbers to decide.

Dense versus mixture-of-experts: throughput and memory

A dense model uses all of its parameters for every token, while a mixture-of-experts (MoE) model has a router that sends each token through only a few expert sub-networks, so it is described by two numbers: total parameters and active parameters per token.

So an MoE with a large total but modest active count can generate faster than a dense model whose total is smaller than the MoE's total but larger than its active count, provided the hardware can hold all of the MoE's weights. When memory is not a constraint and throughput is the objective, compare dense size with MoE active parameters. When a memory budget is stated, compare it with MoE total parameters first: a model whose total weights do not fit is eliminated, however small its active count.

mlflow.genai.evaluate() is the MLflow 3 evaluation harness: you pass data (the evaluation records), optionally a predict_fn (the app to run) and a list of scorers, and it creates an evaluation run in the experiment in which every record gets a trace with the scorers' assessments attached.

Each record can carry three fields:

• inputs: a dictionary with the request. Its keys are the parameter names of predict_fn, because MLflow calls predict_fn(**inputs). A record {"inputs": {"question": ..., "region": ...}} therefore needs def predict_fn(question, region); a function that takes one dictionary, omits a key or uses different parameter names fails or receives the wrong values.
• outputs (optional): an answer the app already produced.
• expectations (optional): ground truth, such as expected_facts, expected_response or per-record guidelines.

import mlflow
from mlflow.genai.scorers import Correctness, Safety

results = mlflow.genai.evaluate(
    data=eval_records,            # list of dicts, DataFrame or evaluation dataset
    predict_fn=my_agent,          # called as my_agent(**record["inputs"])
    scorers=[Correctness(), Safety()],
)

Data can be a list of dictionaries, a pandas or Spark DataFrame, or a Unity Catalog evaluation dataset. To evaluate an agent that is already deployed, without loading it locally or writing a REST wrapper, use mlflow.genai.to_predict_fn("endpoints:/<endpoint-name>"), which turns a Model Serving endpoint into a predict function (an apps:/<app-name> URI does the same for a Databricks App). The URI names a running endpoint or app, so the evaluation calls the deployed version exactly as clients do; loading a registered model into the notebook evaluates a local copy instead.

Score what already exists when you want to judge answers that were already produced; re-run the app with a predict_fn when you want to judge a version's own answers to a set of requests. The data you pass decides which mode you are in.

Two consequences follow. First, stored answers from a model that can no longer be called are scored by passing inputs, outputs and any expectations with no predict_fn; leaving outputs out leaves nothing to judge, and adding a predict_fn would replace the stored answers. Second, to compare a new version on requests captured in production traces, rebuild the records from each trace's request as inputs, drop the trace column, and keep the predict_fn: scoring the trace column would only evaluate the old version again.

Also remember what stored data lacks. A table or CSV of questions and final replies has no execution trace, so a scorer that inspects tool calls or retrieval has nothing to read; if the app is still available, re-running it with a predict_fn produces traces to inspect.

Traces turn a failing score into a located defect: because mlflow.genai.evaluate traces the app on every record and attaches each scorer's assessment to that trace, opening the trace of a failed record shows every span, such as the retriever's query and returned chunks, the tool chosen with its arguments and output, and the final LLM generation.

Aggregate means and a judge's rationale tell you that a record failed and roughly why; the spans tell you where. For a RAG agent that fails Correctness, check whether the required fact is in the retriever span's outputs:

• Fact missing from the retrieved chunks, answer consistent with them: the defect is in retrieval (chunking, query, index contents). Generation-side fixes cannot supply context that was never retrieved.
• Fact present in the chunks, answer ignores or contradicts it: the defect is in generation (prompt, model, parameters).
• Wrong tool called, or a tool returned an error or bad data: the defect is in tool selection or the tool itself.

The rule is to fix the stage the spans point at: more measurement repeats the diagnosis, and the evaluation run's own traces are where its records live.

Comparing two versions

Each evaluate call is its own evaluation run. For a visual, request-by-request comparison, select two runs in the experiment's Evaluation runs view and choose Compare: each request is listed across the runs with its traces and judge assessments, which a non-coding stakeholder can read. To find regressions programmatically, fetch each run's traces with mlflow.search_traces(run_id=...), pair them by their inputs and compare each judge's value per request. Inputs are the stable key because every run creates new traces with new IDs. Run-level means can hide regressions because improvements on some requests offset losses on others.

A judge requires ground truth when it needs to know what the right answer contains, which it reads from the record's expectations; judges that only relate the request, the response and the retrieved context can run on a dataset that holds nothing but questions, because the trace already has everything they read.

The near-twins are worth fixing in memory: Sufficiency needs to know the answer to say whether the context covers it, while Groundedness and Relevance only compare things the trace already holds. The tool-call judges read the tool spans. ToolCallCorrectness is listed by Databricks among the judges that require ground truth, because it compares the calls made with the expected tool calls in the record (open-source MLflow can also run it without them, judging the calls against the request). ToolCallEfficiency needs no ground truth: it looks for redundant or wasteful calls, not for a required order. How the retrieval judges score individual chunks is covered in the retrieval-evaluation lesson.

Practical planning follows from the table: when labeling budget is limited, spend it on the records you will score with Correctness or RetrievalSufficiency, and start immediately with the judges that need no labels. ExpectationsGuidelines sits between the two groups: it needs no reference answer, but someone must still write rules into each record.

Guidelines or ExpectationsGuidelines: global or per-record rules?

Use Guidelines when the same rules apply to every response, and ExpectationsGuidelines when the rules differ from record to record.

Putting per-question rules into one global Guidelines judge, or into one judge per category, makes the judge apply every rule to every response it sees, which produces false failures wherever a rule does not apply. Both are pass/fail judges; neither produces a custom numeric scale. Behavioural rules belong in a guidelines judge because the other built-in judges have fixed criteria: Safety checks harmful content and Correctness matches facts.

A code-based scorer is a Python function decorated with @scorer (from mlflow.genai.scorers) that MLflow calls once per record; it is the right choice for checks that are exact, deterministic and free of judge-model cost, such as valid JSON, a required field, a length limit, a regular expression or a tool-call order.

Parameters are filled by name

Declare only the parameters you need: inputs (the record's request), outputs (the app's response), expectations (the record's ground-truth dictionary) and trace (the full execution trace). A scorer that reads expectations["expected_sku"] therefore needs that key in every record's expectations. Nothing is passed per record when you list the scorer in scorers.

import json
from mlflow.entities import Feedback
from mlflow.genai.scorers import scorer

@scorer
def format_checks(outputs):
    data = json.loads(outputs)          # an uncaught error becomes Feedback(value=None, error=...)
    return [
        Feedback(name="has_order_id", value="order_id" in data),
        Feedback(name="under_150_words", value=len(str(data).split()) < 150),
    ]

Return values

• A primitive (bool, int, float, str) or a single Feedback becomes one metric, named after the Feedback's name or, if none, the function.
• A list of Feedback becomes several metrics, and each Feedback in the list needs its own distinct name.
• The number of metrics is decided by the number of Feedback objects returned, not by what a value contains.

Errors

If the scorer raises an exception for a record, MLflow catches it, records a Feedback whose value is None with the error details, and continues with the other records. One bad record therefore shows up as an error on that record, never as a failed check or a stopped evaluation. You may also catch the error yourself and return a Feedback carrying the error.

When the requirement concerns how the app reached its answer (which tools ran, in what order, whether retrieval returned anything), only the trace holds the evidence, so the scorer declares the trace parameter and searches its spans.

from mlflow.entities import SpanType
from mlflow.genai.scorers import scorer

@scorer
def identity_before_refund(trace):
    tools = [s.name for s in trace.search_spans(span_type=SpanType.TOOL)]
    if "issue_refund" not in tools:
        return True
    return "verify_identity" in tools and tools.index("verify_identity") < tools.index("issue_refund")

trace.search_spans(span_type=SpanType.RETRIEVER) returns the retriever spans; a span's inputs hold what it was given (the query) and its outputs hold what it returned (the documents), so an empty-retrieval check reads the outputs. Watch edge cases: all() over zero matching spans is True, so decide explicitly what a trace without the span should score.

A Guidelines judge or a scorer that reads only outputs cannot see tool calls when the final reply looks the same either way. A trace-aware LLM judge can see them, but it is an LLM call: it costs tokens and is not deterministic, so for exact order or presence checks the code scorer wins.

make_judge creates a custom LLM judge from natural-language instructions that reference template variables, plus a feedback value type; use it when the assessment is subjective (a rubric, tone, appropriate use of evidence) and none of the built-in judges' fixed criteria fit.

from typing import Literal
from mlflow.genai.judges import make_judge

compliance = make_judge(
    name="compliance",
    instructions=(
        "Read the customer request in {{ inputs }} and the reply in {{ outputs }}. "
        "Classify the reply against the bank's disclosure policy."
    ),
    feedback_value_type=Literal["compliant", "non_compliant", "needs_review"],
)

feedback_value_type controls the shape of the result: bool for pass/fail, int or float for a numeric scale (for example a 1 to 5 rubric), Literal[...] for a fixed set of categories enforced through structured output, simple dicts and lists, and str for free text. If you leave it unset, the judge decides the type itself, so MLflow recommends setting it explicitly. Only Literal constrains the value to named categories: free text relies on the instructions alone, so labels can drift ("Compliant."), and numeric or boolean types carry no category names. Every judgement comes back as Feedback with a rationale.

Choosing between scorer types

Subject-matter experts improve an agent by recording their judgements on traces without writing code, through the Review App; those judgements then become ground truth for evaluation datasets or human labels for aligning LLM judges.

Expectations versus feedback

Labeling sessions: reviewing existing traces

A labeling session is a run in the experiment that holds the traces to review, the label schemas that define the questions (built-in schemas such as expected facts and guidelines, or custom feedback and expectation schemas) and the assigned reviewers. Experts work in the Review App; their answers are saved as assessments on each trace. When the goal is ground truth for Correctness, use the expected-facts schema, then call session.sync(to_dataset=...): sync matches records on inputs, overwrites the same-named expectations of matching records, adds unmatched traces as new records and leaves untouched records as they were. Syncing is a merge keyed on inputs.

Chat testing: experts talk to the agent

Before any traces or dataset exist, experts can converse with a deployed agent. Attach a Model Serving endpoint with get_review_app() and review_app.add_agent(agent_name=..., model_serving_endpoint=...), then share the Review App URL ending in /chat. Every conversation and rating is captured as a trace. Chat reviewers need three things: an identity provisioned in the Databricks account, the workspace access entitlement for the workspace hosting the experiment (provisioning alone is not enough) and CAN_QUERY on the serving endpoint. Grant the least permission that lets them query the endpoint.

Alignment makes an LLM judge apply the experts' standard by calling the judge's align() method on traces that carry both the judge's assessment and human feedback; it returns a new judge, which you then use (and can register) in place of the original, with no hand-editing of the instructions.

Alignment works on built-in judges and on judges created with make_judge. It needs:

• Matching names. The human feedback's assessment name must exactly match the judge's name. Feedback logged under a different name (for example a labeling schema called "quality" for a judge named "answer_quality") is ignored, and the aligned judge behaves like the original.
• Feedback, not expectations. Alignment reads human feedback on the judged traces; expected facts are a different label type.
• Enough examples. At least 10 labeled traces is the minimum, and 50 to 100 work better.
• Balance. Include poor and fair answers so that a meaningful share of labels, around 30%, are negative; a set where every label is "good" teaches the judge nothing about what experts reject. The sample must include traces the judge passes as well as fails, or its false passes stay hidden.
• Real reasoning. Experts' own, specific rationales carry the standard the judge learns from.

The principle: a judge learns a team's standard only from that team's labeled examples, which is what alignment supplies.