Databricks GenAI Engineer: CI/CD for Agents, Prompt Lifecycle, MCP Servers and Agent Interfaces

CI/CD for a Databricks agent means that reviewed code, not hand-copied artifacts, reaches each environment automatically, and that every environment builds its own data assets (indexes, endpoints, jobs) from its own data. On Databricks the standard vehicle is a bundle: Declarative Automation Bundles, the current name for Databricks Asset Bundles. A bundle describes the agent code, the jobs that build its Vector Search index and log the model, and the serving endpoints, with one target per environment (dev, staging, prod), each pointing at its own workspace and catalog.

Two promotion patterns exist, and the data constraint decides between them:

Folder copies inside one workspace, or exporting notebooks and importing them by hand, fail both goals: they are not automated and they do not separate environments.

How the pipeline authenticates

The CI system (for example GitHub Actions running bundle deploy on merge to main) should run as a service principal, never as a person, and should avoid long-lived secrets. Databricks supports workload identity federation: you create a federation policy on the service principal that trusts the GitHub Actions OIDC token issuer, the workflow presents its short-lived OIDC token, and Databricks exchanges it for a short-lived Databricks token. Compare the alternatives: an OAuth client secret stored in GitHub is still a long-lived credential, and a personal access token both is long-lived and runs as a person.

Testing an individual component means exercising one piece of the agent (a tool, a parser, the retriever, the routing logic) directly with fixed inputs and asserting its output, without running the whole agent or calling a live model. These tests are fast, cheap and deterministic, so they can run on every pull request and fail only when that component's logic is wrong.

• Tools. A Unity Catalog function or Python tool can be called directly with known arguments; assert the result (for example, that a refund calculation rounds correctly).
• Post-processing and routing. Replace the LLM client with a stub that returns fixed responses, including malformed JSON, and assert that the parser and the tool-routing code handle each case. The stub removes token cost and randomness and lets you test failure formats on purpose.
• Retrieval. Call the retriever directly with a set of known questions and assert that the expected chunk IDs come back. This isolates retrieval from generation and costs no LLM tokens.

The rule for a unit test is that it must be automated, deterministic, cheap and isolated to one component. Anything that adds model randomness, spends tokens, depends on a live endpoint or needs a person to look at the result fails that rule, however useful it is elsewhere. Full evaluation with LLM judges, live endpoint calls, manual exploration and trace review all have their place, at other layers:

Evaluation gates and post-deploy smoke tests

An evaluation gate evaluates one exact, immutable candidate and promotes that same candidate only if its scores pass. The identity matters as much as the threshold: production must receive exactly what was evaluated.

• Pin the candidate. Evaluate a prompt by its version number (or a model by its version), then point the production alias at that same number. Resolving @latest once for evaluation and again for promotion races against teammates who register new versions in between.
• Gate before exposure. Moving the production alias first and evaluating afterwards exposes users to untested text, even if you move it back on failure.
• Do not re-register on promotion. Copying the evaluated template into a new version creates a version that was never evaluated.

A smoke test catches what offline evaluation cannot. A serving endpoint reporting Ready only means its container started. Problems that come from the serving environment, for example a Vector Search index or other resource that was not declared when the agent was logged, so the endpoint's identity cannot reach it, surface only when the deployed endpoint handles a request. After Ready, send a few representative requests to the staging endpoint and fail the pipeline unless valid answers return. Anything that runs outside the endpoint, under a different identity, cannot see this class of fault; only a request through the deployed endpoint exercises the endpoint's own credentials and declared resources.

A pipeline that rewrites the source table must also make the index pick up the change, and how depends on the index's sync mode. (Databricks has renamed Mosaic AI Vector Search to Databricks AI Search; the exam guide uses the older name. Endpoint types and sync modes in depth are covered in the Mosaic AI Vector Search lesson.)

Change Data Feed on the source table is a prerequisite for Delta Sync, not a trigger: enabling it does not make a Triggered index refresh. In Triggered mode the only thing that moves new rows into the index is a sync request; operations on the surrounding infrastructure do not.

Storage-optimized endpoints, chosen for cost at very large scale, support only Triggered sync, and each sync partially rebuilds the index. So the right design is to call sync() as the final step of each load, which refreshes exactly when new data lands. A separate timer that syncs every few minutes wastes rebuild work between loads, and moving to a standard endpoint for Continuous sync gives freshness the requirement did not ask for at a higher cost.

Changes that need a new index: build, wait, switch, delete

An index's schema and embedding model are fixed when it is created. Adding a column (for example a region column to filter on) or moving to a different embedding model cannot be done by syncing the existing index. The zero-downtime pattern is:

1. Change the source table if needed (add the column).
2. Create a new index with the new schema or embedding model.
3. Wait for its initial sync to finish, so it is fully populated.
4. Switch the agent (configuration or alias in your code) to the new index.
5. Delete the old index.

Deleting first and recreating under the same name causes an outage; switching before the initial sync completes serves partial results. Direct Access upserts do not apply to a Delta Sync index and never change a schema.

The MLflow Prompt Registry (prompts stored in Unity Catalog, such as main.support.answer) gives prompts the same lifecycle as code: every registered version is an immutable snapshot, and aliases are mutable pointers to versions. That split is what makes promotion, rollback and reproducible tests possible. (Template syntax, registering versions and loading by alias in chain code are introduced in the LangChain and prompt engineering lesson; this section is about lifecycle.)

Two consequences follow. A deployed agent should load a named alias, never @latest: with @latest, any new registration (an experiment, a draft) becomes what production loads, and nothing attached to a version as metadata can stop that. And a regression suite that must reproduce a past baseline should load the version number that the baseline run logged; aliases move with releases, and a new alias created today points at today's text, not the baseline's.

Versions also carry lineage: a commit message when registered, and tags such as the author or ticket. These are metadata for people; they never affect which version a URI resolves to.

Prompts are promoted across environments by giving one registered prompt an alias per environment (for example dev, staging and production) and moving the next alias to a version that passed testing. The same application code runs everywhere; each environment's configuration names the alias it loads, so promotion needs no code change and no redeploy. Any scheme that encodes the environment somewhere else (in the prompt name, in code, or in metadata looked up at runtime) reinvents what aliases already do and usually needs a deploy to change.

A promotion step therefore has two parts: resolve the source alias to its version number (a loaded prompt exposes .version), then call mlflow.genai.set_prompt_alias() with the prompt name, the target alias and that number. For example, promoting an evaluated dev candidate to staging for a prompt named main.hr.policy_qa:

cand = mlflow.genai.load_prompt(
    "prompts:/main.hr.policy_qa@dev")
mlflow.genai.set_prompt_alias(
    "main.hr.policy_qa", alias="staging",
    version=cand.version)

Watch the direction: the alias you read is the source, the alias you set is the target. Promotion never registers anything: a version created during promotion is by definition a version that was not evaluated.

Rollback

Because the previous version still holds exactly the text that worked, rollback is one set_prompt_alias() call that points production back to it. An alias-loading app picks it up without a redeploy, no new version is created, and the bad version stays available for investigation. The principle is to move a pointer, not data: aliases never fall back on their own, and any rollback that removes versions or aliases, or creates new versions, either breaks loads, loses evidence or ships text that was never evaluated.

Check the automation too. If a scheduled job promotes whatever staging points to, and staging still references the bad version, the next run undoes the rollback. Move staging off the bad version as well, or pause the job.

mlflow.genai.load_prompt() caches what it loads, so an alias move reaches running replicas only when their cached copy expires. By default an alias-based load is cached with a 60-second TTL (configurable through the MLFLOW_ALIAS_PROMPT_CACHE_TTL_SECONDS environment variable), while a version-number load is cached with no TTL, because a version can never change.

• To make alias moves propagate faster, pass cache_ttl_seconds to load_prompt(), for example 5 seconds. Most requests are still served from cache.
• cache_ttl_seconds=0 bypasses the cache, so every call queries the registry: instant propagation, but a registry lookup on every request.
• The TTL is the lever: anything that forces propagation by redeploying, or that stops loading by alias, gives up the no-redeploy benefit aliases exist for.

So a lag of about a minute after moving an alias is expected behaviour, not a deployment fault.

The Model Context Protocol (MCP) gives agents a standard way to discover and call tools. On Databricks you pick the server type by asking who already provides an MCP server for the capability:

For data questions, the current Databricks guidance points natural-language business questions at a Genie-based MCP server and specific SQL at the Databricks SQL MCP server. Typical traps: rebuilding something Databricks already provides duplicates a managed server; registering a plain REST API as an external MCP server fails because external registration only works for a server that already speaks MCP; calling a vendor MCP URL with a token from a Databricks secret keeps the token out of code but bypasses Unity Catalog governance. For an API described by an OpenAPI specification, Databricks provides an OpenAPI MCP server template for Databricks Apps that reads the spec (for example from a volume) and takes its credentials from an HTTP connection, which is far less code than hand-writing one tool per operation, whether as MCP tools or as Unity Catalog functions.

MCP registration is evolving: Databricks now manages registered MCP servers through Unity Catalog (Unity Gateway), and endpoint URL patterns change. The decisions above are stable; specific URLs are not.

External servers: the Unity Catalog HTTP connection

An external MCP server is registered through a Unity Catalog HTTP connection, which stores the server's endpoint and credentials so they never appear in agent code. The connection's authentication type decides whose identity reaches the vendor:

When each person may see only what they can see in the vendor system, and each call must be attributable to them, only per-user OAuth works. Filtering by an email the agent passes along is not enforcement.

Who needs which privilege on a registered MCP

When an external server is registered in Unity Catalog as an MCP (the current Unity Gateway path), the MCP itself is the securable that callers use. Permissions split by role:

Keep USE CONNECTION away from end users: it permits direct use of the connection, which bypasses the tool selection and policies defined on the MCP, so granting it to callers is the over-grant, not the least privilege. Ownership or MANAGE goes further still, letting a principal alter the MCP or grant it on. None of these grants changes record-level access inside the vendor; that is the authentication type's job.

Legacy path: before MCPs became Unity Catalog securables, agents called an external server through a per-connection proxy URL, and there USE CONNECTION on the connection was what callers needed. You may still meet this older pattern in existing projects. Likewise, a custom app built from the OpenAPI MCP server template reads its credentials from an HTTP connection, so that app's callers need USE CONNECTION on it.

Custom servers on Databricks Apps

• Transport: Streamable HTTP. Databricks Apps hosts web servers, so the stdio transport (a local subprocess) does not apply.
• Listening: bind to 0.0.0.0 on the port in DATABRICKS_APP_PORT; the platform routes traffic only to that address and port.
• Name: start the app name with mcp- (for example mcp-ticket-tools) so AI Playground recognises it as an MCP server.
• URL: clients use the app URL plus the server's MCP path, typically /mcp; the bare app root lists no tools.
• Permissions: callers need CAN USE on the app; CAN MANAGE adds editing, redeploying and stopping it. Each resource type has its own permission vocabulary: app permissions govern apps, Unity Catalog privileges govern securables such as connections and MCPs, and serving-endpoint permissions govern endpoints.

The right interface is the one that fits who is using the agent, at what stage, and where they already work:

Direction of the call matters. A tool integration (for example an MCP server for a chat platform) lets the agent call out to that platform; it is not an interface, because nothing carries users' messages in to the agent. A chat interface inside a messaging platform needs a bot registered with that platform: the platform delivers user messages to the bot, and the bot calls the agent's serving endpoint and replies where the user asked. Requirements such as "users must stay in the tool they already use" therefore rule out anything that hands them a link to somewhere else.

Every interface decides whose identity reaches the agent and its tools, and that decides which rows a Unity Catalog row filter returns.

Databricks Apps

Each app has its own service principal, whose credentials the platform injects. With app authorization, you declare the serving endpoint as an app resource with the lowest permission that works, Can query for an app that only sends chat requests (Can view under-grants, Can manage over-grants), and reference it from app.yaml. No personal access token belongs in the app's code or configuration.

App authorization means every query runs as the app's service principal, so every user sees the same rows. When each user must see only their own rows, enable user authorization with the model-serving scope so the app forwards each signed-in user's token to the endpoint, and build the agent for on-behalf-of-user authorization (declare the user API scopes its tools need when logging it, and create tool clients with the user's credentials). Both layers are required: forwarding the token does nothing if the agent's tools still run as the endpoint's service principal. Changing how much any identity may do never changes which identity runs the query; only these two layers do.

Microsoft Teams (Public Preview)

The documented pattern is an Azure Bot Service bot (its code hosted for example on Azure App Service) with a Microsoft Entra ID app registration and an OAuth connection that signs users in with Entra ID. To call the agent as that user, create a Databricks OAuth federation policy that trusts the tenant's token issuer and the bot app's audience; the bot then exchanges the user's Entra ID token for a Databricks token for the same user. Add on-behalf-of-user authorization in the agent and tool queries run with that person's permissions. The test for any design is the same: if the token that reaches Databricks belongs to one shared identity, every user gets that identity's rows, whatever user details travel alongside in the request and whatever is logged.