Databricks GenAI Engineer: Choosing LLMs and Embedding Models

To choose an LLM, turn the application's attributes into requirements, drop every model that fails a hard requirement, then pick the cheapest and fastest model that still clears the quality bar. The best-sounding model on a leaderboard is often the wrong answer because it breaks exactly one stated constraint: it is too slow, too expensive, missing a capability, or hosted somewhere the data may not go.

Work through the attributes in this order:

1. Hard requirements (pass/fail): languages, tool calling, structured output, context window, where prompts may be processed, licence, whether you must fine-tune the weights, and release status (some teams allow only generally available features in production).
2. Quality bar: the minimum accuracy or task metric the application needs, measured on your own evaluation data.
3. Objective: among the models that pass 1 and 2, minimise what the business cares about, usually cost per request or latency.

Use the smallest model that meets the measured quality bar. Per-request cost and latency grow with model size, so when candidates score within noise of each other on your task, the smaller instruct model wins. For most workloads the model's per-token price is the dominant cost, which is why the choice of model usually moves the bill far more than adjustments to prompt wording or output limits.

Base the decision on evidence: compare the candidates on a representative evaluation set drawn from real traffic, not on a few hand-picked prompts or a vendor demo.

One-off costs versus per-request costs

Every option for closing a quality gap has a cost profile. One-off costs are paid once: fine-tuning a model, building an evaluation set, curating training data. Per-request costs are paid on every call: the model's token price and the number of input and output tokens each request carries. At high volume, per-request costs dominate the total, so the economical path is usually the one that spends once to let a small model meet the bar rather than one that adds cost to every request for the life of the application.

Fine-tuning fits that pattern when the task is narrow and repetitive and you have plenty of labelled examples: the model learns the task's format and vocabulary instead of being reminded of them in every prompt. For broad, open-ended tasks with little labelled data, fine-tuning has less to learn from, and a stronger general model may be the better buy.

The context window must hold everything in the request plus the longest reply you allow: system prompt, instructions, retrieved chunks, the user's question (and any conversation history you resend), and the maximum output tokens. Input and output share one window, so counting only the input undersizes it.

Work it out as a sum. A summarisation request that carries a 40,000-token call transcript, 1,500 tokens of instructions and up to 3,000 output tokens needs at least 44,500 tokens of window. A model with a 32K window fails even before the reply is counted; a model with a 64K window fits with room to spare.

Then buy no more window than you need. Among candidates that fit, extra window adds no value on its own, and longer-context or larger models usually cost more, so the requirement is a threshold, not a score to maximise.

Recheck the sum whenever the request grows: more retrieved chunks, longer history or a longer system prompt all consume window. When input plus max_tokens exceeds it, requests fail or replies are cut short. Resolve that by asking which quantities are fixed requirements (the amount of context, the reply length) and which can give; if none can give, the model's window has to grow.

Capabilities are pass/fail filters: a model that lacks one the application needs is out, however well it scores elsewhere. Check them on the model card or the serving documentation before any quality comparison.

• Function (tool) calling: required when an agent must invoke APIs to read or change data. Without it the agent can only talk, not act.
• Structured outputs: on Databricks Foundation Model APIs you pass a JSON schema through the response_format parameter, and decoding is constrained so every reply conforms to that schema. Instructions and examples in the prompt only encourage a format; nothing enforces it. Use structured outputs when code or a typed table consumes the reply directly.
• Languages: the model must be trained for every language your users write in. Quality in an unsupported language is poor regardless of speed or size.
• Modality: image or audio input is a requirement only when the application actually has such inputs.

Choose a reasoning model only when the task needs multi-step reasoning, a test shows it raises accuracy, and the application can absorb its extra latency and tokens. Reasoning models spend a thinking phase, billed as tokens, before the visible answer. That improves results on problems with dependent steps (calculations that build on each other, multi-hop analysis) but delays the first visible word and raises the token bill.

The thinking phase is part of what the model does, so its latency and token cost come with the choice of model or mode rather than with serving settings. Treat a reasoning model like any other candidate: it has to earn its extra cost with a measured accuracy gain on your own evaluation set.

Where a model runs and whether you control its weights are requirements in their own right. Data residency is decided by where inference runs: if prompts and responses may be processed only inside your Databricks workspace, or you will fine-tune the model later, choose an open-weights model served on Databricks Model Serving. If a proprietary model is allowed and wins your evaluation, serve it through an external model endpoint.

Base versus Instruct

Open model families usually ship a base version (trained only to continue text) and an Instruct (chat) version tuned to follow instructions and hold a conversation. Databricks recommends Instruct versions for deployment and base versions for fine-tuning. Deploy Instruct for an assistant that must follow instructions out of the box. Start from base for continued pretraining on unlabelled domain text or for your own instruction tuning; training an Instruct model on raw text would also erode the tuning it already has.

A base model has no concept of roles or instructions: it treats the whole prompt as a document and predicts what text would come next. Served as an assistant, it therefore tends to keep writing in the style of its input rather than answer it, which is how a mistakenly deployed base version usually shows itself.

An embedding model's maximum input length must cover the longest unit you embed: the largest chunk or document, and the longest query users send. Each chunk is embedded in one call, and queries are embedded by the same model, so both sides count. Text beyond the window cannot be represented in the vector.

The generally available Foundation Model APIs embedding endpoints differ mainly in input length:

Match the embedded unit to the expected queries

A vector represents the overall meaning of the text it was computed from. The longer and more varied that text, the less any single detail contributes to it. The optimisation strategy follows from what users ask:

• Precise lookups answered by a sentence or a paragraph favour short, focused passages, so the matching passage's vector is about that detail and the result points straight at it. A long window lets you embed bigger units, but bigger units blur details rather than sharpen them.
• Questions about a document as a whole, where each document must stay one index row, need a window that holds the longest document. Anything past the window is silently dropped, so content near the end of a long document would never influence its vector.
• Documents longer than any available window can still be represented whole by embedding a condensed representation of each document, such as a generated summary, and keeping the full text alongside for the LLM.
• Long queries, when users paste large blocks of text, put the query side over the limit too. Because queries and documents must share one model, a longer-input model on the query side means re-embedding the documents with it as well (see the next section).

Chunk sizing for a given embedding model is covered in the data preparation lessons; this topic runs the other way, choosing the model for given units and queries.

Queries and documents must be embedded by the same model, so switching embedding models means re-embedding the whole corpus and changing the query path together. Vectors from different models live in different spaces even when they have the same number of dimensions, so similarity between a query vector from one model and a document vector from another is meaningless. A half-finished switch shows up as retrieved chunks that look unrelated to the questions.

Treat a model replacement on a self-managed index as one migration: re-embed every chunk with the new model, write the new vectors to the index (or build a new index and cut over), and switch the query embedding at the same moment. On a Delta Sync index with Databricks-computed embeddings, changing the embedding endpoint likewise means rebuilding the index so every vector comes from the new model.

Language fit

The embedding model must cover the languages of both the documents and the queries. Check the model card's language list first: an English-focused model maps other languages poorly however long its input. For cross-language retrieval, where a question written in one language must find a passage written in another, choose a multilingual embedding model, which places text with the same meaning near each other whatever the language. Embedding and generation are separate roles, so the multilingual requirement applies to the embedding model itself, not only to the LLM that writes the answer.

When no hosted embedding endpoint meets every requirement (language, input length, release status, licence), an open embedding model that does can be logged, registered in Unity Catalog and served on a custom model serving endpoint.

Databricks surfaces models in three places: Databricks-provided models in system.ai, partner and vendor models on Databricks Marketplace, and models you bring from external hubs such as Hugging Face.

Model cards

A model card is a screening tool: use it to drop models that fail a hard requirement before spending time on experiments. Read every field against the application:

• Task type and intended use: the model's role must match the job (a generative instruct model for writing and extraction, an embedding model for vectors).
• Languages: the users' and documents' languages must be listed.
• Context length: must hold the input plus the reply (for an LLM) or the longest chunk and query (for an embedding model).
• Licence: must permit the intended use.

A card describes the model in general. Anything that depends on your data, your prompts or your serving setup has to be measured in experiments. Screen on the card; decide on experiments.

Licences

Filter by licence first, then rank by score; a licence is a cheap pass/fail check that saves evaluation effort.

For a derivative model (a fine-tune or other model built from another), check the whole licence chain, not just the latest label: the terms of the model it was built from continue to bind what was built on it.

Pick the metric that measures the failure the business cares about, compare every candidate on the same evaluation set, filter by the requirements, then rank by the objective. Log each candidate as a run in one MLflow experiment so metrics, parameters and costs sit side by side.

With imbalanced classes, overall accuracy is dominated by the common classes, so a model can score well while handling a rare class badly. Look at the per-class metric for the class that matters: recall when missing a case is the costly error, precision when false alarms are. Perplexity is an early, cheap signal of how familiar a model already is with your domain's language; the task metrics still decide.

Thresholds filter, the objective ranks

Treat each requirement (a minimum task score, a maximum p95 latency, a cost ceiling) as a filter: discard every run that fails any of them, then rank the survivors by the objective, such as lowest cost. A run that leads on one metric is still out if it breaks another requirement.

Is the difference real?

• Same evaluation set: scores are comparable only when the candidates were evaluated on the same examples. Runs on different or older versions of a set measure different things.
• Enough examples: on a small set, one or two examples swing the score by several points, so small gaps are within noise. Weigh a gap against the size of the set, and invest in a bigger representative set when the decision is costly or long-lived.

LLM judges and scorers for qualities such as groundedness and safety belong to the evaluation and monitoring material; this topic covers choosing a model from the classic task metrics your experiments produce.