Databricks CI/CD: Git Folders, Declarative Automation Bundles and the Databricks CLI (Data Engineer Associate)

A Databricks Git folder (formerly Databricks Repos) is a workspace folder linked to a remote Git repository, and the Git dialog is where you branch, commit, push, pull, merge, rebase and reset without leaving the workspace. New work starts on a new branch: in the Git dialog choose Create Branch, give it a name and pick the base branch (usually main). The folder switches to the new branch, and work stays off main until it has been reviewed.

Switching branches is done from the branch dropdown in the same dialog. Three behaviours surprise people:

• Uncommitted changes carry over. Edits you have not committed travel to the branch you switch to and show there as uncommitted changes, unless they conflict with that branch. Git folders do not stash them for you. To keep the other branch clean, commit or discard first.
• Missing assets are deleted, then recreated with new IDs. If the target branch does not contain a notebook or file, switching removes it from the workspace. Switching back recreates it as a new object with a new ID and URL, and that cannot be undone, so bookmarks, shared links and anything else built on the old URL break.
• The branch belongs to the folder, not the person. A Git folder has one checked-out branch for everyone who opens it. If several engineers share one Git folder (for example under /Workspace/Shared) and one of them switches branch, everyone else's view changes. The recommended pattern is for each developer to clone the repository into a Git folder under their own user folder and branch there; production jobs read code from Git or from a bundle deployment, never from someone's working folder.

Git operations run against the provider (GitHub, GitLab, Azure DevOps, Bitbucket and others) using the Git credential each user links in their own Databricks settings (a personal access token or the provider's app/OAuth integration). Provider permissions alone are therefore not enough: a user with write access in the provider still cannot push from a Git folder until they have linked their own credential. Workspace permissions on the folder do not replace it.

To share work, open the Git dialog, review the highlighted changed files, enter a commit message (it is required) and choose Commit & Push. That single action commits the changes and pushes them to the remote branch, where teammates and reviewers can see them. Pull moves changes the other way, from the remote branch into your folder.

Notebook outputs. Notebooks in a Git folder are stored in source format by default (a .py or .sql file with cell markers), which holds code only, so Commit & Push never sends cell results. To version results in Git, the notebook must be stored as IPYNB and output commits must be enabled for it; running the cells or clearing state does not change what a source file can hold.

Pull requests live in the Git provider. Git folders handle cloning, branching, committing, pushing, pulling, merging, rebasing and resetting; the review happens in the provider. The flow is:

1. Create a feature branch from main in the Git dialog (uncommitted edits made on main carry over to it).
2. Commit & Push to the feature branch.
3. Open a pull request from the feature branch to main in the Git provider; a reviewer approves it and it is merged there.

Most teams protect main in the provider: direct pushes are rejected and every change needs an approved pull request, so merging into main from the Git dialog or pushing from another account does not get around review. A pull request follows its source branch, not fixed commits: to address review comments, make the edits on the same branch and Commit & Push again, and the new commits appear in the existing pull request with the discussion intact. A new branch would need a new pull request.

Git folders offer four ways to change a branch's history, and the right one depends on whether you want to keep commits and whether anyone else shares the branch. The key detail is that rebase and reset both force push.

Merge and rebase both bring main's new commits into a feature branch; the constraint decides. Rebase rewrites commits that teammates may already have pulled, so on a shared branch use merge. When a linear history is required and nobody else has pulled the branch, rebase is the only operation that puts the feature's commits on top of main. Reset is destructive by design: it is the tool for making an experimental branch match main again, not for integrating changes.

Conflicts. When a merge or rebase hits conflicts, the UI lets you resolve each file by editing it, or choose Keep all current or Take all incoming, then Mark As Resolved and Continue Merge (or Continue Rebase). Abort is the escape hatch: it cancels the operation and undoes everything it changed, returning the folder to where it was before the merge or rebase began. It is the way out of any merge or rebase you no longer want to finish.

Declarative Automation Bundles (formerly Databricks Asset Bundles) are Databricks' infrastructure-as-code format for CI/CD: a project folder whose YAML configuration describes Lakeflow Jobs, Lakeflow Spark Declarative Pipelines and other workspace assets alongside the code they run, kept in source control and deployed with the Databricks CLI to each environment. They solve the drift that comes from clicking jobs together separately in dev, test and prod workspaces. Git folders move code; bundles move code and the job and pipeline definitions.

Every bundle has one databricks.yml at its root. The top-level mappings you need to recognise:

Two near-twins are easy to confuse. The top-level include lists additional YAML files that form part of the bundle configuration, the usual way to split a large databricks.yml into one file per job or pipeline. sync.include is about uploads: on deploy the bundle uploads project files but skips anything matched by .gitignore, so sync.include adds back a generated file that must stay out of Git but is needed in the workspace, and sync.exclude drops further files.

To start a project, databricks bundle init asks a few questions and writes a starter bundle from a template (databricks.yml with targets, a sample job and code). Not every workspace object is a bundle resource: cluster policies, for example, are created by admins outside the bundle and are referenced by ID or lookup.

Bundle variables are how one configuration carries values that differ by environment. Declare a variable once under the top-level variables mapping (optionally with a default), reference it anywhere in the configuration as ${var.<name>}, and set its value per environment under each target's variables mapping:

variables:
  catalog:
    description: Catalog the jobs write to
    default: dev_sales
targets:
  test:
    variables:
      catalog: test_sales
  prod:
    variables:
      catalog: sales

The job definition then uses ${var.catalog} once and never needs a copy per target. Substitutions in ${...} are resolved by the CLI at deployment. Do not confuse them with the {{...}} dynamic value references that Lakeflow Jobs resolves at run time (such as {{job.id}}); the bundle leaves those alone and passes them through to the job.

A target's values apply only to that target. Deploying the test target never reads values set under prod; if test sets nothing, the variable's default is used.

Complex variables. Ordinary variables hold single values such as strings or numbers. Declaring type: complex lets a variable hold a mapping or a list, for example a whole new_cluster definition (Spark version, node type, autoscale range, Spark configuration). Each job then references ${var.etl_cluster} in its own job_clusters entry: every job still gets its own job cluster, and one edit to the variable changes them all. A string variable cannot stand in for nested YAML.

Precedence. When a variable gets values from several places, the CLI uses the first source that provides one, in this order:

1. The --var option on the command, for example databricks bundle deploy -t test --var="catalog=qa_override"
2. An environment variable named BUNDLE_VAR_<name>, for example BUNDLE_VAR_catalog
3. The target's .databricks/bundle/<target>/variable-overrides.json file
4. The target's variables mapping in the configuration
5. The variable's default

Two consequences follow. A CI runner that exports BUNDLE_VAR_... silently beats both the overrides file and the target mapping, so editing those has no effect while the environment variable is set. And the only way to beat an environment variable for one run without touching the runner is --var on the command itself.

Lookup variables and built-in substitutions supply environment-specific values without anyone typing them in. Lookup variables handle an object created outside the bundle (a cluster policy, SQL warehouse, cluster, instance pool, service principal, alert, dashboard, job, metastore, pipeline or query) that has the same name but a different ID in each workspace. Instead of recording each ID under its target, declare:

variables:
  wh_id:
    lookup:
      warehouse: finance-sql

and reference ${var.wh_id}. At deployment the CLI resolves the name to the ID in whichever workspace the target deploys to, and fails if no single object matches. Writing per-target IDs also works but must be copied and maintained, and putting a display name where an ID is expected does not work at all.

Alongside custom and lookup variables, bundles provide built-in substitutions that need no declaration. The ones that matter most:

Choose by what should vary. Per-environment values come from ${bundle.target} (if names follow the target name exactly) or from a variable set per target. Per-developer values in a shared dev workspace come from ${workspace.current_user.*}, which is also how development mode builds its user prefix; ${bundle.target} and ${bundle.name} are identical for every developer.

For references between resources in the same bundle, use ${resources...}. It resolves at deploy time to the object that deployment manages, in any workspace and under any development-mode prefix, including on the first deploy to a new workspace. A lookup variable is the wrong tool there: it must find an existing object by name before deployment, so it fails when the job does not exist yet and misses prefixed dev copies. Use lookups only for objects the bundle does not deploy.

A target is a named deployment environment under targets, and settings inside a target override or extend the top-level configuration for that target only. This is how the same codebase is promoted from dev to test to prod while each environment keeps its own settings.

• Default target. Exactly one target may set default: true; bundle commands run without -t use it. Neither mode: development nor presets make a target the default.
• Separate workspaces. When dev, test and prod are different workspaces, set workspace.host under each target. Then databricks bundle deploy -t prod goes to the prod workspace with no --profile option; credentials still come from the environment, and the CLI can match a configured profile by host. A single top-level host cannot differ per environment.
• Resource overrides merge. A target can declare the same resource key as the top level and list only the fields that differ. Lists merge by key: job tasks by task_key and job clusters by job_cluster_key. A prod target entry for job sales_rollup with a job cluster keyed shared holding only a larger node_type_id and num_workers changes just those settings; every task that uses shared still comes from the top-level definition unchanged.

targets:
  prod:
    resources:
      jobs:
        sales_rollup:
          job_clusters:
            - job_cluster_key: shared
              new_cluster:
                node_type_id: i3.2xlarge
                num_workers: 8

Using a new key in the override would add a second cluster that no task uses, and repeating the whole job under the target defeats the point of a shared definition.

A target's mode sets a bundle of behaviours suited to personal development or to production.

Presets (such as name_prefix, trigger_pause_status, jobs_max_concurrent_runs, pipelines_development, tags) adjust these defaults, but they apply to every resource in the target. A resource's own setting wins over mode defaults and presets, so to keep one job's hourly schedule running in dev while everything else stays paused and prefixed, set pause_status: UNPAUSED on that job's schedule in the dev target.

run_as. By default, jobs run as the identity that deployed them, so production jobs fail when that person's account is deactivated. Databricks recommends run_as with a service principal (service_principal_name) for production. Set it in the prod target, not at the top level: a top-level run_as applies to every target, so dev jobs would run with production data access, while target-level run_as leaves dev jobs running as whoever deployed them. At the top and target level, run_as accepts a user or a service principal but not a group. A recommended setup separates the deploy identity from the run identity; for a deployer to set run_as to another service principal it needs the CAN_USE permission (Service Principal User role) on that service principal, which is far narrower than admin rights.

Production mode without a service principal. Production mode is designed for service principals. When a user deploys instead, it adds checks so production assets do not belong to one person: root_path, file_path, artifact_path and state_path must not point at a specific user's folder (use a shared path such as /Workspace/Shared/.bundle/prod), and both run_as and permissions must be specified.

The Databricks CLI manages bundles through the databricks bundle command group, and almost every command takes -t <target>.

Deploy tracks resources by ID, not name. That has two consequences. First, configuration that is not bound creates a new object, so adopting a hand-built job is a two-step move: databricks bundle generate job --existing-job-id <id> to write the YAML, then databricks bundle deployment bind <key> <id> so deploys update the original job and keep its ID and run history. Second, a deploy deletes previously deployed resources that were removed from the configuration. To stop managing a job while keeping it running in the workspace, run bundle deployment unbind for it first, then remove it from the YAML. Unbind is the opposite of destroy: destroy removes what the bundle deployed; unbind leaves the object in place.

The Databricks CLI is not only for bundles: command groups such as databricks workspace, databricks jobs, databricks pipelines and databricks repos manage workspace objects directly, for example updating a Git folder to the latest commit from CI. A CI/CD pipeline drives the same CLI commands non-interactively, and a typical design ties each stage to a Git event:

• On every pull request: databricks bundle validate to catch configuration and schema errors cheaply, then bundle deploy -t test and bundle run -t test <key> so the change is proven in test before a reviewer approves it.
• On merge to main: bundle deploy -t prod. The merge is the only event that reaches prod, which pairs naturally with a production-mode target that checks the Git branch.

Deploy flags for automation:

Bundle deploys hold a deployment lock, stored with the deployment state in the workspace (not in the local .databricks cache), so two deploys to a target cannot collide. A cancelled deploy can leave the lock behind; once you have confirmed nothing else is deploying, one run with --force-lock recovers. --force does not touch the lock, and neither flag changes which resources a deploy removes.

CI identity. Databricks recommends a service principal rather than a user for CI/CD. Workload identity federation is the most secure option: the CI system (for example GitHub Actions) presents its own short-lived OIDC token and receives access as the service principal, so no Databricks secret is stored in the CI system. OAuth machine-to-machine (M2M) also authenticates a service principal but requires storing its client secret. Variable values for CI can come from BUNDLE_VAR_* environment variables or --var, following the precedence above.