Databricks Platform Architecture, Delta Lake, Unity Catalog and Compute (Data Engineer Associate)

The Databricks Data Intelligence Platform is a lakehouse: data is stored once, in open-format Delta Lake tables on low-cost cloud object storage, and given the reliability and performance of a data warehouse, so the same tables serve BI dashboards, SQL analytics, data engineering and machine learning under one governance layer, Unity Catalog.

The lakehouse answers a specific problem. Many companies ran two systems: a data lake (cheap, open files such as Parquet, read directly by data scientists, but with no transactions and coarse governance) and a data warehouse (reliable, fast SQL, but proprietary storage) fed by a copy of the lake. Two copies drift out of sync, dashboards lag the lake by the load schedule, and access rules are kept in two places. The lakehouse removes the second copy instead of making the copy faster.

Two building blocks carry the design. Delta Lake turns files in object storage into transactional tables, and Unity Catalog governs every way of reaching them, from SQL warehouses to notebooks. On top of these, compute is separate from storage: you pick the compute that suits each workload and they all read the same tables.

Databricks splits into a control plane, which runs in the Databricks account, and a compute plane, where your data is processed. Where the compute plane runs depends on the kind of compute.

• Control plane (Databricks account): the backend services Databricks manages, such as the web application, the job scheduler, cluster management and notebook/workspace configuration. It does not execute your queries.
• Classic compute plane (your cloud account): classic clusters, including all-purpose compute, job compute and classic and pro SQL warehouses, are launched in a virtual network inside your own cloud account. Your network rules, private connectivity to on-premises systems and custom networking apply here.
• Serverless compute plane (Databricks account): serverless SQL warehouses and serverless compute for notebooks, jobs and pipelines run in compute that Databricks owns, in the same region as the workspace, inside a network boundary for each workspace. You do not manage the instances.

Workspace storage. A classic workspace is created with a storage bucket (or container) in your cloud account that holds the workspace's own system data: notebook and query results, job run results, cluster logs and similar internal data, and it can hold the default workspace catalog. It is separate from the storage locations that hold your Unity Catalog tables and volumes, it is not a backup of them, and Databricks documents that it must not be deleted or modified. Serverless workspaces use Databricks-managed default storage instead.

When a compliance question is phrased in terms of "whose cloud account" and "which network", answer by compute type: classic means your account and your network; serverless means the Databricks account in the workspace's region.

A Delta table is a directory of Parquet data files plus a transaction log in the _delta_log subdirectory, and the log, not the directory listing, defines what the table contains. Delta is the default table format on Databricks.

Each write is a commit: a numbered JSON file in _delta_log that records which data files were added and removed and any schema or metadata change. Periodic checkpoint files (Parquet) summarise the log so readers do not replay every JSON file. A table version is the set of files the log says are live after a given commit.

This design gives Delta its ACID guarantees:

• Atomic commits. A writer writes new data files first and then tries to commit. If the job fails before the commit, the files it wrote are referenced by no version, so readers never see them. A failed overwrite leaves the complete previous version visible, not a mix and not an empty table.
• Snapshot isolation. A query reads one committed version from start to finish, even while another job is writing.
• Row-level DML. DELETE, UPDATE and MERGE work by rewriting the affected files and committing the swap, which plain Parquet cannot do. A privacy request to delete one customer's rows is a DELETE statement.

Two consequences follow. First, never edit a Delta table by adding or removing files in storage by hand: the log still references removed files (later reads fail) and one data file usually holds many unrelated rows, so deleting it deletes them too. Changes go through SQL or the DataFrame API so they are committed. The same goes for the old "write to a staging folder, then rename or swap it over the live path" trick: on cloud object storage a directory rename is a copy-and-delete of many objects, not an atomic commit, so readers can see missing or partial files mid-swap, and it still gives you no row-level DELETE. Second, files written by failed jobs or replaced by later commits stay in storage until VACUUM removes them.

Existing Parquet data can be converted in place with CONVERT TO DELTA, which builds a transaction log over the existing files.

Because the log keeps every version, Delta can read and restore past versions: this is time travel, and how far back it reaches is set by two retention properties and by how VACUUM is run.

• DESCRIBE HISTORY table lists versions with their timestamp, operation and user.
• SELECT * FROM table VERSION AS OF 41 or TIMESTAMP AS OF '2026-10-01' reads an old version without changing the table.
• RESTORE TABLE table TO VERSION AS OF 41 (or TO TIMESTAMP AS OF) makes the current table match an earlier version. It does this by adding a new commit, so the restore itself appears in the history and can be undone.

DESCRIBE HISTORY sales.orders;
SELECT count(*) FROM sales.orders VERSION AS OF 41;
RESTORE TABLE sales.orders TO VERSION AS OF 41;

VACUUM deletes data files that the current version no longer references and that are older than the retention threshold, 7 days by default (the table property delta.deletedFileRetentionDuration, or a RETAIN clause). Delta refuses a shorter retention unless a safety check is turned off, because running jobs might still need the files.

Time travel to a version needs both its commit in the log and the data files it referenced:

So with defaults and a weekly VACUUM, a 20-day-old version fails even though its commit is still in the log: its data files were vacuumed. To keep N days of time travel while still running VACUUM to control storage, set delta.deletedFileRetentionDuration to at least N days and keep delta.logRetentionDuration at least N days. Stopping VACUUM protects history but storage grows; OPTIMIZE compacts files but does not delete old ones; a clone does not stop VACUUM on the source from removing the source's files.

Delta schema enforcement (schema validation) rejects any write whose columns or types do not match the table, and because writes are atomic, a rejected write commits nothing. Changing the schema has to be asked for explicitly.

If an append brings a column the table lacks, the default result is a schema mismatch error, not silent dropping of the column and not automatic addition. The ways to change the schema differ in scope and in what they can change:

Pick by scope. When only one job should accept a new column and other jobs on the same cluster should keep failing on surprises, use the write option, not the session or cluster setting. When a column's type must change (for example STRING to BIGINT after casting in the DataFrame), mergeSchema is not enough: rewrite the table with overwrite mode plus overwriteSchema. overwriteSchema has no effect on an append.

df.write.mode("append").option("mergeSchema", "true").saveAsTable("crm.orders")
df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable("crm.customers")

Column mapping (delta.columnMapping.mode = 'name') is a separate feature that allows renaming and dropping columns as metadata changes; it does not make appends accept new columns. Schema evolution in streaming ingestion with Auto Loader is covered in the ingestion lessons.

Unity Catalog is the single governance layer for data and AI assets on Databricks: it holds the metadata, permissions, lineage and audit for tables, views, volumes, functions and models across every workspace attached to it.

Metastore. The metastore is Unity Catalog's top-level container. There is one metastore per region, and every workspace in that region can be attached to it. Attached workspaces see the same catalogs, schemas, objects and grants (unless an admin binds a catalog to specific workspaces). A workspace cannot attach to a metastore in another region, so a company with workspaces in two regions has two metastores.

Three-level namespace. Objects are addressed as catalog.schema.object, for example prod.silver.orders. A shorter name is completed from the session's current catalog and current schema, set with USE CATALOG and USE SCHEMA. Order matters: USE CATALOG also resets the current schema to default, even if it names the catalog you were already in. For example, if a session runs USE SCHEMA staging and then USE CATALOG dev, its current location is dev.default, not dev.staging: the schema choice was discarded by the later catalog switch. Fully qualified names avoid the question entirely and are the safer habit in production code.

USE CATALOG prod;      -- current schema is now prod.default
USE SCHEMA silver;     -- now prod.silver
SELECT * FROM orders;  -- reads prod.silver.orders

Lineage. Unity Catalog captures lineage automatically at runtime from queries run on Unity Catalog objects, at table and column level, with no tracking code in the pipelines. The lineage graph in Catalog Explorer (and the lineage system tables) shows upstream and downstream tables plus the notebooks, jobs and dashboards that read or write them, which is how you find what a column change will affect. Do not confuse it with neighbouring features:

Volumes. A volume is a Unity Catalog object inside a schema that stores non-tabular files of any format (images, PDFs, CSV, JSON, libraries). Files stay files, are read and written by path under /Volumes/<catalog>/<schema>/<volume>/, and are governed by Unity Catalog privileges, the same model as tables. Databricks recommends volumes over DBFS mounts (/mnt paths), which Unity Catalog does not govern. Loading files into binary table columns changes them from files into rows, and workspace files are for code and small project assets, not governed data.

Managed versus external tables and privileges such as GRANT and REVOKE belong to the governance domain and are taught there.

Databricks compute comes in four families, and the right one depends on whether the work is interactive or scheduled, SQL-only or code, and whether it must run in your own network.

All-purpose versus job compute. A scheduled production job running on an always-on all-purpose cluster pays the higher rate all day. Moving it to a Lakeflow Jobs task on job compute cuts cost without changing code, because the cluster exists only for the run and is billed at the lower jobs rate. Auto termination or autoscaling on the all-purpose cluster trims waste but keeps the higher rate.

Sharing job clusters. Each task in a job can get its own new job cluster, but a job can also define a job cluster at the job level that several tasks share. For many short tasks in sequence, separate clusters spend most of the run starting up; one shared job cluster starts once, keeps classic networking and job pricing, and terminates when the run ends.

Photon is Databricks' vectorized query engine. It is on by default in SQL warehouses and serverless compute and can be enabled on classic clusters to speed up SQL and DataFrame workloads.

Serverless compute for notebooks, jobs and pipelines removes cluster management: Databricks provisions, scales and patches the compute, it starts quickly, and you pay for what runs. The trade-off is a narrower feature set, because serverless runs your code through Spark Connect.

Limitations to know:

• DataFrame and SQL APIs only; no RDD APIs. Calls such as spark.sparkContext.parallelize() or rdd.map() fail. The fix is to rewrite them with DataFrame operations; no Spark config or library turns RDDs on.
• No R or Scala in notebooks. Python and SQL are the notebook languages; Scala runs only as JAR tasks in jobs.
• No init scripts and no compute policies. Libraries are declared in the environment: the notebook's Environment panel, or for a job task the task environment's Dependencies (requirements.txt-style entries from PyPI, workspace files or volumes). A job that relied on an init script to install packages fails on import until those packages are added as dependencies. A notebook task uses the notebook's own environment by default.
• %pip still works, but it is not where a job's packages belong. A notebook-scoped %pip install at the top of a notebook does run on serverless, but it reinstalls on every run and hides the job's requirements inside per-run notebook code. The rule for production jobs is to declare packages once in the environment (the Environment panel or the job environment's Dependencies), so every run and every task gets the same versions without code changes.
• Only a limited set of Spark configurations can be set.
• No Spark UI; use the query profile to inspect performance.

Networking. Serverless runs in the Databricks account, not in your VPC, so workloads that must reach systems only through your own network design (for example an on-premises database over private connectivity) are a reason to stay on classic compute.

Cost modes for serverless jobs. Serverless jobs and pipelines offer performance optimized mode, which starts and runs faster for time-sensitive work, and standard mode (performance optimized turned off), which uses fewer DBUs but can take several minutes longer to start. A batch job with hours of slack and a cost goal fits standard mode; an SLA-tight job fits performance optimized.

A SQL warehouse is compute dedicated to SQL, and Databricks offers three types that differ in where they run, how fast they start and which performance features they include. Databricks recommends serverless where it is available.

Predictive IO speeds up selective scans and other reads. If a workload needs Predictive IO and Photon but must run in your own cloud account, pro is the only type that meets all three conditions. A warehouse's scaling settings (minimum and maximum clusters) handle concurrency peaks; serverless also removes the minutes-long cold start.

Access mode decides who can use a classic cluster and how Unity Catalog permissions are enforced on it.

So: several analysts sharing a cluster with per-user permissions means standard mode. A team that needs R or the ML runtime and accepts shared permissions uses a dedicated cluster assigned to their group (group assignment needs Databricks Runtime 15.4 LTS or above); assigning it to one person would lock the others out, and serverless does not run R.

Cost model. Databricks bills in DBUs (Databricks Units) per unit of time, at rates that depend on the compute type (jobs compute cheaper than all-purpose, separate rates for SQL warehouses and serverless). Classic compute adds the cloud provider's charge for the virtual machines. Every billable record lands in the system.billing.usage system table, with fields such as sku_name, usage_quantity and usage_metadata.job_id (filled for job compute and serverless jobs), so DBUs per job per day is a schedulable SQL query; joining to list prices turns DBUs into cost. The jobs UI, cluster event logs and system.compute.clusters describe runs and configuration, not DBU quantities.

Levers on classic compute, each solving a different problem: