Databricks GenAI Engineer: Model Serving Access, Foundation Model APIs, ai_query() and Agent Memory

Access to a Model Serving endpoint is controlled by permissions on the endpoint itself, at three levels that each include the one below: CAN VIEW, CAN QUERY and CAN MANAGE. Least privilege means giving each identity the lowest level that still lets it do its job.

Two points cause most mistakes. First, seeing an endpoint is not the same as querying it: CAN VIEW never permits sending a request, so every identity that submits requests needs CAN QUERY. Second, Unity Catalog privileges on the registered model do not decide who can call the endpoint. EXECUTE on the model or USE SCHEMA on its schema governs the model asset; the endpoint has its own access control list. Rolling back to an earlier version or changing a traffic split is a configuration update, so it needs CAN MANAGE, while an app that only sends questions needs CAN QUERY and nothing more.

A served chain gets credentials for outside systems through endpoint environment variables that reference Databricks secrets, written exactly as {{secrets/<scope>/<key>}}. When the endpoint is created or updated, Model Serving fetches the secret and exposes its value to the model as an environment variable, so the key never appears in plain text in the endpoint configuration, the model artifacts or the code.

• The syntax is literal. A value such as secrets/translate/api_key without the double braces is just a plain-text string, not a secret reference.
• The creator needs READ on the secret. The secret is fetched with the identity that creates or updates the endpoint, so that identity needs READ on the scope. If it lacks it, the create or update fails. WRITE or MANAGE on the scope is more than required.
• Callers never touch the secret. People and apps that query the endpoint need CAN QUERY on the endpoint, not access to the scope.
• Never package keys as artifacts or plain-text variables: both leave the key readable to anyone who can see the model or the configuration.

Inside the model, code reads the value like any other variable, for example os.environ["TRANSLATE_API_KEY"], typically in load_context when the client is created.

A deployed agent reaches Databricks resources (LLM endpoints, Vector Search indexes, Genie spaces, tables, functions) in one of three ways: automatic authentication passthrough, on-behalf-of-user (OBO) authorization or manual authentication. Passthrough and OBO are taught in depth in the Agent Framework lesson; this section covers when you must fall back to manual authentication and what that changes.

For manual authentication Databricks recommends OAuth credentials of a service principal (machine-to-machine) for production, so access does not depend on a person's account; a personal access token is simpler but suited to development. Supply the values as secret references in the endpoint's environment variables (DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET), never as plain text. If the agent loads prompts from the Prompt Registry, the service principal also needs CREATE FUNCTION, EXECUTE and MANAGE on the schema that holds the prompts.

The trap: overriding these security environment variables turns off automatic passthrough for every other resource the agent uses. After the switch, the index search and the LLM call also run as the service principal, so that principal needs access to them too, or calls that worked before start failing with permission errors. Passthrough only ever covers the resource types it supports; anything else needs credentials you supply.

AI Gateway rate limits cap how much traffic a serving endpoint accepts, so one caller cannot exhaust a shared endpoint while everyone keeps access. Limits are set in the endpoint's AI Gateway settings, not in code, and can be expressed as queries per minute (QPM) or tokens per minute (TPM).

• Endpoint limit: a ceiling on all traffic to the endpoint combined.
• Default user limit: applies to each caller individually, which is the tool for stopping a single runaway notebook or script.
• Custom limits for a specific user or service principal: override the default for that identity only. An app backend that calls as one service principal for hundreds of people counts as a single caller, so it needs its own higher limit rather than a looser default for everyone.
• Group limits: shared by the members of a group.

TPM limits apply to endpoints that serve foundation or external models; custom model and agent endpoints accept QPM limits only. To cap the tokens an agent consumes, put the TPM limit on the model endpoint the agent calls. Rate limits are the control that caps how much an identity uses while leaving its access in place.

Foundation Model APIs serve Databricks-hosted foundation models on Model Serving in two modes: pay-per-token, ready-made endpoints you can call immediately, and provisioned throughput, endpoints you create with dedicated capacity.

A fine-tuned model registered in Unity Catalog cannot be called through the base model's pay-per-token endpoint; deploy the registered version to a provisioned throughput endpoint.

Databricks also offers a priority pay-per-token tier that it recommends for latency-sensitive production traffic. Provisioned throughput remains the recommended choice for production workloads that need high throughput, performance guarantees or fine-tuned models, and it is the mode documented for HIPAA compliance.

When the model belongs to a third-party provider (OpenAI, Anthropic and others), use an external model endpoint instead. It routes requests to the provider using the provider key stored in a Databricks secret, gives every team one governed entry point under endpoint permissions, and supports AI Gateway features such as rate limits and usage tracking. Calls are billed by the provider under your own contract, unlike Databricks-hosted models, which Databricks bills. The endpoint is the single governance point: the key lives in one secret, and every call passes through endpoint permissions and AI Gateway controls.

Databricks serving endpoints for chat, completion and embedding models are OpenAI-compatible, so the OpenAI SDK works with three settings: set base_url to the workspace URL plus /serving-endpoints, authenticate with a Databricks token, and pass the serving endpoint name as model.

from openai import OpenAI

client = OpenAI(api_key=token,
    base_url=f"{host}/serving-endpoints")

reply = client.chat.completions.create(
    model="databricks-meta-llama-3-3-70b-instruct",
    messages=[{"role": "user", "content": q}])

vectors = client.embeddings.create(
    model="databricks-gte-large-en",
    input=chunks)

• The model value is the endpoint name exactly as listed on the Serving page, not a Hugging Face identifier or a bare model name.
• /api/2.0/serving-endpoints is the management REST API (create, update, list), not the OpenAI-compatible query path.
• Use the API that matches the task: embeddings.create returns vectors; chat.completions.create and completions.create return generated text.

The same endpoints can also be called over REST, with the Databricks SDK or MLflow deployments client, and from SQL with ai_query().

To serve an LLM application, register the logged chain or agent in Unity Catalog and deploy that version to a Model Serving endpoint. The endpoint exposes the chain over REST with autoscaling, and the chain, running on the endpoint, calls the Foundation Model API endpoint and any retrievers it uses. Deploying the whole chain keeps its retrieval and prompt logic on the server, behind one governed, real-time endpoint.

For agents, agents.deploy() from the databricks-agents package deploys a Unity Catalog model version in one call, for example agents.deploy("main.support.agent", 3). Beyond creating the endpoint, it provisions short-lived credentials for the resources declared at logging, sets up the Review App where subject-matter experts give feedback, and connects MLflow tracing. Creating the endpoint directly with the SDK gives you only the endpoint. Databricks documentation now also offers newer tooling for brand-new agents, but agents.deploy() remains the documented route for a registered version. Logging and registering the chain is covered in the pyfunc and Unity Catalog lesson.

ai_query() is a built-in SQL function that applies a serving endpoint's model to many rows in one query, which makes it the tool for batch inference: the inputs already sit in a table, the volume is large, and nobody waits on a single answer. Examples are nightly summaries of stored transcripts, labelling new tickets in a scheduled job and backfilling sentiment over years of reviews. When a person or a transaction waits for the reply (a chat assistant, a voice agent, a fraud check at checkout), use a real-time endpoint instead.

SELECT id,
  ai_query(
    'databricks-meta-llama-3-3-70b-instruct',
    CONCAT('Summarise: ', transcript),
    modelParameters => named_struct('max_tokens', 256),
    failOnError => false) AS result
FROM support.calls

Databricks recommends submitting the full dataset in one query and letting the platform parallelise the requests, rather than looping over small batches.

Where it runs and who can run it. ai_query() is not available on Classic SQL warehouses; run it on a serverless or pro SQL warehouse, or on compute with Databricks Runtime 15.4 LTS or above (the current reference states the 15.4 LTS floor and the Classic exclusion; failOnError and responseFormat themselves need that runtime level). Because it is a built-in function of the runtime, the runtime version is the gate; libraries installed on the cluster do not change it. The identity running the query needs CAN QUERY on the endpoint, plus the usual SELECT on the input table.

A Model Serving endpoint is stateless across requests: it may run several replicas, any request can land on any of them, and replicas restart on scale-to-zero or redeploy. History kept in a Python attribute inside predict therefore lives in one process only, so state that appears to persist in a single development process does not survive a scaled deployment. Earlier turns must come from the request or from an external store.

The rule: per-turn agent state needs fast, transactional reads and writes by exact key. Stores built for analytics, similarity search or static files do not provide that, which is why Lakebase sits on the request path and Delta sits behind it for analysis.

Short-term memory belongs to one conversation and is keyed by its session ID; long-term memory (preferences, facts about the user) spans conversations and must be scoped by the person. Take that identity from the authenticated request context, never from a name the LLM extracts from the conversation or that the user types. Scoping by that user ID does two jobs at once: each person's memories are recalled in every conversation, and they never surface in anyone else's.

Access to the store follows the identity that runs the agent. In a notebook the agent uses the developer's own credentials, so reads and writes succeed; once deployed it typically runs as a service principal, and every store it reads or writes must grant that principal the matching access. Grants must name the identity the agent actually runs as, with each kind of access (read, write) it performs.