Databricks GenAI sample questions — 25 answered and explained

When a label set is too large to put in a prompt, use embeddings to retrieve a few candidate labels, then let a generative model classify against that short list.

Two facts decide this. First, 3,000 labels cannot be listed in one prompt, so the LLM must see only a subset. Second, the requirement wants an LLM to make the final decision. Embedding retrieval (top-k candidate codes per title) solves the first, and an LLM choosing among the retrieved candidates satisfies the second. Pure nearest-neighbour skips the LLM, batching the taxonomy multiplies cost and needs reconciliation, and fine-tuning on one description per code has too little data.

1. Note the constraint: the full label set does not fit in a prompt.
2. Use embeddings to shortlist the most similar codes for each title.
3. Pass only the shortlist, with descriptions, to the LLM for the final choice.
4. Check that the design keeps the LLM as the decision-maker.

Exam tip: Too many labels for a prompt: retrieve candidates with embeddings, then classify with an LLM.

Decompose a requirement into steps and give each step to the right tool: models for language tasks, plain code for fixed business rules.

The requirement mixes language tasks (reading the email, writing a reply) with a deterministic business rule that must be auditable. Extraction supplies the model and date, code applies the rule exactly, and generation drafts the reply around a decided outcome. Letting an LLM apply the rule, classify eligibility directly, or do everything in one prompt would usually give a plausible answer, but none is reproducible or auditable.

1. Split the requirement into its steps: read inputs, decide, respond.
2. Mark which steps need language understanding and which are fixed rules.
3. Give the fixed, auditable rule to deterministic code, not a model.
4. Order the steps: extract, check, generate.

Exam tip: Use models for language and code for rules; an auditable decision belongs in deterministic logic.

Map each requested output to its own model task; one business requirement often needs more than one task.

The managers asked for two outputs. Competitor names are spans copied from the text, and some are not on any list yet, so they must be extracted as entities rather than classified against a fixed list. The recap is a condensed version of the transcript, which is summarization. Sentiment is not requested, and a follow-up email is a different deliverable with a different audience.

1. List each output the business asked for.
2. Match each output to a task: names to extraction, recap to summarization.
3. Check whether a fixed label list can cover the values: new products mean it cannot.
4. Drop tasks that produce something nobody asked for.

Exam tip: One task per required output; don't add tasks nobody asked for.

Databricks AI Functions include task-specific functions such as ai_mask, ai_extract, ai_classify and ai_summarize; each one maps to a single model task.

The requirement is to keep the transcript but hide specific entity types, which is masking. ai_mask does that with a list of labels. ai_extract is its near-twin but returns the entities instead of hiding them, ai_classify only flags rows, and ai_summarize changes the content without guaranteeing removal.

1. Identify the transformation: keep the text, hide named entity types.
2. Match it to the task-specific function that masks entities.
3. Reject functions that extract, flag or condense instead.

Exam tip: Hide entities in place: ai_mask. Pull them out as fields: ai_extract.

Repeated page furniture (headers, footers, page numbers) is extraneous content that degrades embeddings and should be filtered before chunking.

The noise is in the data, so the first fix is in the data: remove text that repeats on every page. Metadata filters, bigger chunks and re-ranking all leave the boilerplate in place and only work around it later in the pipeline.

1. Identify content that repeats on every page and carries no answer.
2. Remove it at parse time, before chunking and embedding.
3. Treat retrieval-side tweaks as secondary to clean input.

Exam tip: Clean before you chunk: strip repeated headers, footers and page numbers.

Extraneous content includes site boilerplate and duplicate copies; both should be filtered before chunking.

The symptoms name two problems: shared page furniture (navigation, banners, footers) and duplicate pages. Removing boilerplate and deduplicating fix them directly. Lowercasing, stripping punctuation and stop words are keyword-search habits that hurt dense embeddings, and dropping tables throws away content.

1. Match each symptom to its cause: repeated boilerplate and duplicate pages.
2. Pick the filter that removes each cause.
3. Reject normalisation steps that remove meaning rather than noise.

Exam tip: Filter boilerplate and duplicates; don't strip text that embedding models need.

Outdated versions are extraneous content: filter superseded documents out of the knowledge base.

The problem is that the index holds contradictory facts. The data-level fix is to keep only the current version, using effective-date or status metadata to decide. Merging files leaves every allowance in the chunks, a version column that nothing filters on still lets old chunks win, and a re-ranker measures relevance, not currency. A prompt instruction is ruled out by the stem and would not remove the conflicting facts from the index either.

1. Recognise conflicting versions of the same document in the source.
2. Decide at ingestion which version is authoritative.
3. Exclude superseded versions instead of hoping retrieval picks correctly.

Exam tip: Stale versions in the index produce stale answers: ingest only the current one.

Source selection: choose authoritative, current and audience-appropriate documents that contain the knowledge the use case needs.

Retrieval cannot improve on its sources. The official guides for the shipping release are accurate, scoped to what customers run and safe to show them. Support tickets and forum threads are tempting because they reflect real questions, but they are unofficial, version-mixed and may expose personal or internal information; they can supplement a knowledge base after cleaning, not form its core. Archived guides for earlier releases introduce outdated answers.

1. State what the assistant must answer and for whom.
2. Prefer sources that are authoritative, current and appropriate for that audience.
3. Treat noisy, unofficial, sensitive or outdated sources as supplements at most, never the core.

Exam tip: Garbage in, garbage out: pick authoritative, current sources for the audience you serve.

UCFunctionToolkit exposes governed Unity Catalog functions to LangChain agents as tools.

The function already exists and is governed in Unity Catalog; the agent only needs to call it. UCFunctionToolkit wraps named functions as LangChain tools, so the logic stays where it is. The other classes target vector indexes, chat endpoints and embeddings.

1. Identify the asset: an existing Unity Catalog function.
2. Identify the need: the agent calls it as a tool.
3. Pick the class that converts UC functions into tools.

Exam tip: UC function to LangChain tool: UCFunctionToolkit.

MLflow Prompt Registry templates use {{double-brace}} variables.

The registry recognises variables written as {{context}} and {{question}}. A template written with single braces has no variables as far as the registry is concerned, so format() returns it unchanged. The load method and argument style are fine, and to_single_brace_format() solves the opposite problem.

1. Read the symptom: placeholders survive format().
2. Recall the registry's variable syntax: double braces.
3. Fix the template, not the load call or arguments.

Exam tip: Prompt Registry variables are {{name}}; single braces stay literal.

Registry templates use double braces; LangChain needs single braces, so convert a copy with to_single_brace_format().

Two facts combine. The registry's syntax is {{var}}, which the batch jobs depend on. LangChain's f-string templates treat {{ }} as an escaped literal brace, so the raw template would not bind variables. to_single_brace_format() returns a LangChain-ready copy without changing the registered version. Re-registering with single braces breaks format(), and pre-formatting defeats invoke-time filling.

1. Note the two consumers and their syntaxes.
2. Recall what doubled braces mean to LangChain.
3. Convert for the new consumer rather than change the shared template.

Exam tip: One registered prompt, two syntaxes: keep {{ }} in the registry and convert with to_single_brace_format() for LangChain.

MLflow Prompt Registry versions prompts in Unity Catalog and lets apps load them by alias.

All three needs map to registry features: prompts are Unity Catalog entities (governance), each register_prompt() call adds an immutable version (history), and aliases let an app load prompts:/catalog.schema.name@alias so the version can change without a code edit. The other options can be built into something similar but only with custom work.

1. List the needs: central versions, UC governance, swap without code changes.
2. Match each need to a registry feature.
3. Prefer the managed feature over a custom store.

Exam tip: Versioned, governed, swappable prompts: MLflow Prompt Registry with aliases.

Aliases are mutable pointers to immutable prompt versions; promotion moves the alias.

The app follows the production alias, so promoting means repointing that alias with set_prompt_alias(). Versions cannot be edited, tags are metadata, and switching to a version URI requires the code change the scenario forbids.

1. See how the app loads the prompt: by alias.
2. Recall that versions are immutable and aliases move.
3. Move the alias rather than touching code or versions.

Exam tip: Promote or roll back a prompt by moving its alias, not by editing versions or code.

A version URI pins an immutable snapshot; only an alias URI follows promotions.

Two facts decide. A prompts:/name/3 URI always returns version 3, whatever any alias says. Of the alias URIs, @latest tracks the newest version, including drafts, whereas a named alias such as production tracks exactly what was promoted. So the app should load @production.

1. Read the load URI: it pins a version.
2. Recall that versions never change, so alias moves cannot reach a pinned load.
3. Choose between alias URIs by what each resolves to: production versus newest draft.

Exam tip: Load by a named alias to follow promotions; a version number pins, and @latest follows drafts.

Key fields in the user's input should drive an exact lookup whose results are injected into the prompt.

The order ID is a key field: it names one record whose current values the answer must state. Extract it, query the row, and add the status and carrier to the prompt as labeled context. Vector similarity is for meaning, not exact keys; fine-tuning freezes changing data; and pasting the whole table does not scale.

1. Identify the key field in the user's input.
2. Fetch the exact, current record for that key.
3. Inject the needed fields into the prompt as labeled context.

Exam tip: An exact key in the question deserves an exact lookup, injected into the prompt.

Post-processing in a pyfunc model enforces output rules for every consumer.

Pre- and post-processing coded in predict travel with the model, so every caller of the endpoint, including ai_query(), gets the same validated output. Prompt instructions are the cheaper option, but they only raise the chance the LLM complies. Doing it in each client duplicates logic and misses SQL callers, and a gateway guardrail can block or filter a response but cannot reshape it into the answer and sources fields.

1. Note the requirement is a guarantee for all callers.
2. Rule out measures that only ask the LLM nicely.
3. Put deterministic checks in the model's post-processing.

Exam tip: If every caller must get it, enforce it in the model's post-processing, not the prompt or the client.

Pre-processing runs in predict before the model call; post-processing runs after it.

Anything that depends on the incoming request belongs in predict. Masking must happen before the endpoint call so the sensitive value never leaves the chain, which is marker (1). Marker (2) is post-processing and is too late. load_context runs once per load and has no request to work on.

1. Decide whether the step depends on the request (predict) or not (load_context).
2. Decide whether it must run before or after the LLM call.
3. Sensitive input must be cleaned before it is sent.

Exam tip: Clean inputs before the LLM call, shape outputs after it, both inside predict.

An LCEL chain pipes template, chat model and output parser in that order.

LangChain Expression Language runs each step on the previous step's output. The prompt template turns {question} into messages, ChatDatabricks calls the Databricks serving endpoint, and StrOutputParser extracts the text. Swapping the order breaks the data flow, and an embeddings class has no generation step.

1. Trace what each step receives and returns.
2. The template must run first, the parser last.
3. Use a chat model class, not an embeddings class, to generate text.

Exam tip: prompt | chat model | output parser: format, generate, parse.

A RAG chain fans the question out to the retriever and to the prompt at the same time.

The dictionary at the start of the chain runs each value on the same input. retriever | format_docs turns the question into a context string, and RunnablePassthrough forwards the question as-is, so the template gets both variables. Dropping the question key or skipping the dictionary leaves the template without one of its inputs, and swapping the keys fills the right variables with the wrong values.

1. List the variables the template expects.
2. Check each one is filled with the right value.
3. Use RunnablePassthrough to forward the original input.

Exam tip: Every template variable needs its own branch: retriever for context, passthrough for the question.

Models from code logs a chain as a Python file plus set_model(), avoiding object serialisation.

Databricks recommends code-based logging (Models from Code) for agents and chains: the chain lives in a .py file that ends with mlflow.models.set_model(...), and log_model receives the file path. MLflow stores the source and rebuilds the chain on load, so unpicklable clients are never serialised and the logged code can be read. Passing any object, wrapped or not, still serialises it.

1. Spot that the failure is object serialisation.
2. Choose the logging style that stores code, not objects.
3. Remember the file must call set_model on the chain.

Exam tip: Log chains from code: a .py file that calls set_model, passed as a path.

Anything in the model's context can be extracted by prompt injection, so secrets must stay out of prompts.

No prompt rule or output filter reliably stops a model from disclosing text it was given. The robust guardrail is to never give it the secret: the tool code reads the key from a Databricks secret and makes the call, and the model only decides when to call the tool. The Safety and PII guardrails target other kinds of content.

1. Assume the system prompt can be extracted, because it already was.
2. Ask what the model actually needs: the ability to call the API, not the key.
3. Move the credential into a secret used by tool code.

Exam tip: If a secret is in the prompt, assume it will leak; keep credentials in tool code via secrets.

Indirect prompt injection arrives in content the agent processes; the controls that hold are those outside the model, on the tool itself.

The email text is untrusted data that the model may treat as instructions. Prompt rules and LLM-based classifiers lower the chance of that, but both depend on a model judging correctly, and security asked for controls that hold even when the model is fooled. The defence must limit what the tool can do: an approval step before sending and a recipient allowlist both cap the damage regardless of the model's decision. Logging is detection, not prevention.

1. Identify the attack path: instructions hidden in an email the agent reads.
2. Apply the stated constraint: the control must work even if the model complies.
3. That excludes anything that relies on the model's judgement.
4. Choose controls on the tool: human approval and a restricted recipient scope.

Exam tip: Assume the model can be fooled; constrain the tools so a fooled model can do little harm.

Each evaluated record produces an MLflow trace whose spans show every intermediate step.

mlflow.genai.evaluate traces the agent on every record and attaches the scorers' assessments to those traces. Opening the trace of a failed record shows each span, such as the tool chosen, its arguments, its output and the final generation, so the failing step can be located. Aggregate metrics and judge rationales describe the outcome, and inference tables cover production traffic.

1. Failures are known; the failing step is not.
2. Each evaluated record has a trace with one span per step.
3. Inspect the spans of the failed records.

Exam tip: Scores say what failed; traces show where.

Per-request regressions are found by comparing assessments on matched traces, not by comparing run-level means.

Each evaluation run stores one trace per record with the judges' assessments attached. Retrieving both runs' traces with mlflow.search_traces(run_id=...) and pairing them by their inputs lets the engineer compare each judge's value per request and list the regressions. Run-level means hide offsetting changes, and trace IDs differ between runs, so they cannot pair the records.

1. The question is per request, not overall.
2. Assessments live on each run's traces.
3. Pair traces across runs by their inputs and compare the values.

Exam tip: Regressions hide in means; compare matched traces' assessments.

Deterministic, rule-based checks belong in code-based scorers.

A function decorated with @scorer receives the output and can parse it as JSON and confirm the field exists, returning True or False with no model call. LLM judges, whether Guidelines, make_judge or Correctness, cost tokens and can vary on borderline cases, which the requirement excludes.

1. The check is mechanical: parse JSON, look for a field.
2. The constraints are no model cost and repeatable results.
3. Use a code-based @scorer.

Exam tip: Exact, mechanical checks: write a @scorer function, not an LLM judge.