Databricks GenAI: Prompt Design, Output Formats and Model Task Selection

Choose the model task from the shape of the output the business needs, not from the input or from which model is available. Read the requirement for what must come back — one value from a known list, specific values copied from the text, a shorter text, the same text in another language, new text, or a measure of closeness — and that output names the task.

Two habits keep the choice honest. First, match what the consumer does with the output: a dashboard that counts needs labels, a payment system needs typed fields, an executive needs prose. Second, add only the tasks someone asked for — sentiment, a follow-up email or a reply draft may sound useful, but a task that produces no required output is cost and risk without benefit.

Use classification when the answer comes from a list you already know, extraction when the answer is text that appears in the input, and summarization when the answer is a condensed restatement. They are easy to confuse because all three "read" a document; they differ in what they return.

Single-label vs multi-label

Decide the label cardinality per output field, from how many values can truly apply. A maintenance report about one machine needs a single-label machine field, while the same report can describe several fault types, so its fault field is multi-label. Each field keeps its own rule: a field where exactly one value applies stays single-label, and a field where several apply keeps all of them so that counts per value stay correct.

Open-ended values are extraction, not classification

Classification can only return labels you gave it. When the set of possible answers is open — every medication named in a clinical note, including drugs that are not on any list you maintain — the values must be extracted as entities from the text. And when a value must be reproduced exactly as written, such as a serial number or a meter reading, define an extraction with one field per value: a summary restates content in new words, so it is not a reliable carrier of exact values.

Summarization and translation

Summarization changes length; translation changes language. A one-page recap of a German maintenance log, written in German, is summarization only; a short Spanish bulletin condensed from long English release notes needs both. Map every hop of a conversation to a language: if staff work in one language and the people they serve write in many, incoming messages need translating into the staff's language and replies need translating back into each sender's language.

Use an embedding model whenever the requirement is about closeness in meaning: semantic search, near-duplicate detection, matching records that are worded differently, or narrowing a very large label set. An embedding model turns each text into a vector once; texts with the same meaning land close together, so a search for "my parcel never turned up" can find an article titled "Missing deliveries" even though they share no keywords.

Embeddings also win on cost at scale. Asking a chat model "do these two records describe the same thing?" grows with the number of pairs, which explodes quadratically across millions of records; embedding each record once and comparing vectors (for example with Databricks AI Search, formerly Mosaic AI Vector Search) grows with the number of items. Only a comparison of meaning catches records that say the same thing in different words — matching on exact text, or on coarse groupings, does not.

Very large label sets: retrieve, then classify

When a taxonomy has thousands of codes, listing them all in one prompt exceeds the context budget. The standard pattern is two stages: embed the inputs and the code descriptions, retrieve the top candidate codes for each input by vector similarity, then let an LLM choose among that short list with the descriptions in view. Each prompt stays small, every input is compared against the whole taxonomy, and the final decision still benefits from the LLM reading the descriptions.

Break the requirement into one task per required output, order them by dependency, and keep fixed business rules in deterministic code rather than in a prompt. A pipeline that "reads expense receipts, applies the per-diem limit and writes an approval note" is really three steps: extract the amount, date and city, apply the limit in SQL or Python, then generate the note from the known outcome.

• Language work goes to models; rules go to code. An LLM applying a numeric or date rule usually gets it right, but its output is not guaranteed to be reproducible, and an auditor cannot check a decision buried in generated text. A code check gives the same answer every time and leaves a trail.
• Classify first when the output depends on the input's type. If lab reports, prescriptions and referral letters each have their own fields and destination table, and nothing upstream records which kind arrived, classify the document kind first, then extract with that kind's schema, then validate its required fields. A single schema that requires every kind's fields pushes the model to fill fields the document does not have.
• Several outputs, several tasks. A field-service visit note that must yield "a priority from four levels" and "the equipment serial number as written" needs classification and extraction; a webinar transcript that must yield "products mentioned" and "a short recap" needs extraction and summarization.

How those tasks are wired together as chain components, tools and agents — and when Agent Bricks does the whole job — is covered in the sibling lesson on chain components, tools and Agent Bricks.

In Databricks SQL, prefer a task-specific AI Function when one returns exactly the output you need, and use ai_query when you must choose the endpoint, write a custom prompt or enforce your own output schema. Task-specific functions need no prompt and no endpoint choice — they run on Databricks-managed models — which makes them the least-effort option for analysts working in SQL.

Near-twins to keep apart: ai_mask hides entities inside the text, while ai_extract pulls them out as values and leaves the text unchanged. ai_similarity returns a number you can sort and threshold; ai_classify returns a label.

What ai_query adds

ai_query is the fit when calls must go to a specific endpoint (for example a fine-tuned or provisioned throughput endpoint the team has evaluated), when the prompt is your own, or when each row must come back in a fixed JSON shape. Its arguments have separate jobs: responseFormat constrains the output to a schema; modelParameters passes settings such as temperature or max tokens; failOnError => false controls error handling, returning a per-row error instead of failing the whole query.

SELECT ai_query(
  'listings-finetuned-endpoint',
  CONCAT('Describe this listing as JSON: ', listing_text),
  responseFormat => '{"type": "json_schema", "json_schema": {
    "name": "listing", "strict": true,
    "schema": {"type": "object",
      "properties": {
        "brand": {"type": "string"},
        "colours": {"type": "array", "items": {"type": "string"}},
        "condition": {"type": "string",
                      "enum": ["new", "used", "refurbished"]}},
      "required": ["brand", "colours", "condition"]}}}'
) AS listing
FROM marketplace.listings;

State the format explicitly and measurably in the prompt: the structure, how many items, how long each is, and what must not appear. "Return a headline of at most eight words, then one sentence, and nothing else" can be checked; "make it punchy" leaves the count and shape open.

• Say what not to output. Chat models are trained to be conversational and often wrap answers in a friendly lead-in or sign-off. When downstream code parses the response, instruct the model to return only the requested object or value — no text before or after it, and no Markdown fences around it.
• Put standing rules in the system prompt. Rules that must hold for every reply of a multi-turn chat belong in the system prompt, which is sent with every request and carries the highest instruction priority. Instructions given once in an early user turn lose influence as the conversation grows, and rules placed inside retrieved passages read as reference material rather than instructions.
• Generation settings are not format controls. max_tokens and stop sequences only cut output off, which can truncate JSON mid-object. Higher temperature increases variation, so format becomes less consistent; temperature 0 makes output more repeatable (near-deterministic, though serving endpoints do not guarantee identical output across calls), but it does not make output correctly shaped.

Reasoning plus a parseable answer

Some decisions are more accurate when the model works through the rules before answering. Ask for the reasoning first, inside delimiters such as <reasoning>…</reasoning>, so it can inform what follows, then the answer inside its own delimiters such as <answer>…</answer>. Downstream code reads only the answer block, which it can find reliably even when the reasoning discusses every option.

Add few-shot examples — sample inputs paired with the exact output you want — when an instruction alone fails on particular cases or when you have approved samples of the target structure. Examples show rather than tell, need no labelled dataset or training, and are the first thing to try before fine-tuning or a larger model.

• Target the failing cases. If most weights convert correctly to grams but phrasings such as "2 lb 3 oz" or "one and a half kilos" do not, include examples of exactly those phrasings with the correct output. Examples aimed at the inputs that fail teach the model what the instruction could not.
• Few-shot before fine-tuning. A handful of approved samples is far too few to fine-tune on, but it is exactly what a few-shot prompt needs.
• Balance the examples. Models copy every pattern in the examples, not only the format — including the label distribution and the length and tone of inputs. If every example is a short label-A input, the model drifts toward label A and toward expecting short inputs. Cover every label and a range of input styles.
• Examples belong in the prompt as examples. Clearly marked input/output pairs are read as a pattern to copy; the same samples supplied as retrieved reference text are read as content to answer questions about.

Databricks Foundation Model APIs and other chat endpoints that support structured outputs can constrain generation through the request's response_format (or ai_query's responseFormat). Use json_schema when you know the object's keys and types, and json_object when you only need valid JSON whose keys cannot be listed in advance.

Support varies by model — some models accept only json_schema — so check the endpoint's model before relying on json_object. Prompt wording and sampling settings rely on the model following instructions; only a schema in the request makes the endpoint enforce the shape.

response = client.chat.completions.create(
    model="my-chat-endpoint",
    messages=[{"role": "user", "content": prompt}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "recipe", "strict": True,
                        "schema": recipe_schema},
    },
)

Designing the schema

• Repeating data is an array of objects. A recipe with a variable number of ingredients needs an ingredients array whose objects each hold name, amount and unit, so any count fits and every row keeps its own fields together.
• Several values from a closed set: type: array with items of type string carrying an enum. That allows zero, one or several values, each from the set. Allowed values written only into a property's description are guidance, not a constraint.
• Let absent values be null. A required field of type string forces the model to output some string, so when the document has no such value (a lease with no break-clause date, say) it produces a plausible one. Declare the type as ["string", "null"] and instruct the model to return null when the value is absent; the schema and the instruction must agree.
• Not every keyword is enforced. Size and length keywords such as minItems or maxLength are not guaranteed by Databricks structured outputs, so validate those in code.

Constrain labels with an enum in the response schema, give the model a fallback label for inputs that fit none of them, and define each label where two labels overlap. Outputs that feed joins, routing tables or counts must come from a closed, enforced set of exact strings.

• Enforce the exact strings. Free-text labels drift in case, wording and synonyms ("Payroll", "pay", "Salary question"), and every variant breaks a join. An enum of the exact label strings in the response schema removes the variation at source, because the endpoint can emit only those values.
• Every closed set needs an "other". Once only enum values can be emitted, an input that fits none of them — an out-of-office reply, a newsletter — still has to receive one of them. Add a fallback label (such as other) to the enum, describe in the prompt when to use it, and route those rows to human review.
• Define the boundaries. When near-twin labels (hardware vs network, say) are chosen inconsistently for the same kind of input, write a short definition of each label that states where the boundary cases belong, and include boundary cases among the few-shot examples.

Describe the pipeline's inputs as what the user or system provides plus what must be retrieved, scoped to exactly what the request needs, and its outputs as the exact shape the consumer needs — including any traceability, handoff or "not covered" signal the business requires. Then let volume and latency decide whether data flows per request or in batch.

Inputs: provided, retrieved, scoped and trusted

• Start from what already exists: if the source material is already text (transcripts, OCR output), the pipeline's input is that text, not the original recording or scan.
• Carry the key that scopes retrieval: a store manager's question plus the store ID lets the pipeline fetch that store's sales and stock records, and nothing else.
• Take identity from a trusted source, such as the authenticated session, and enforce it in the retrieval filter itself. Anything typed into the conversation is untrusted input, and an access rule is only as strong as the place it is enforced.

Outputs: the consumer's shape, plus what the business must verify

• Return exactly what the consumer will act on: for building inspectors, each defect with its location and severity; for a field technician, a short briefing rather than raw records.
• Traceability: when every answer must be checkable, include the passage-level identifiers (chunk IDs, sections or pages) that the answer relied on, so a reviewer can go straight to the supporting text.
• "Not covered" is the model's judgement. A similarity search returns its nearest passages for almost any question, so "no results" rarely happens. Make the model state explicitly, as an output field, whether the retrieved text answers the question.
• Structured handoffs: when some requests go to a human, the output is either an answer or a structured record — request type, the relevant identifiers, the user's message — that a work queue can act on.

Volume and latency shape the I/O

One goal can need both paths: a marketplace that tags each new listing while the seller is still on the page needs request–response serving, while ten years of archived listings belong in a one-off batch job — and both should share one output schema so new and historical results stay consistent. Streaming does not make generation faster; it shows the first words immediately, which is what users perceive as responsiveness when full answers take seconds.