Configure Microsoft Fabric Workspace Settings: Spark, Domains, OneLake, Airflow

Fabric workspace settings live at three levels, and most questions on this objective are really asking which level owns a control. A workspace is the container where a team creates items — lakehouses, warehouses, notebooks, pipelines, eventhouses — and every workspace gets its compute from a capacity, the pool of purchased Fabric compute (an F SKU or a trial) that one or more workspaces share.

The pattern to remember: the tenant decides whether a feature exists, the capacity decides how much compute freedom workspaces get, and the workspace admin decides how the team uses it. A symptom is usually solved by routing it to the right level: a missing "New pool" option is a capacity setting; an item ignoring its own compute settings is a workspace setting; nobody in the tenant being able to create an Airflow job is a tenant setting.

Spark workspace settings decide which pool backs the notebooks and Spark job definitions in the workspace, and the core choice is between the pre-warmed starter pool and a custom pool you shape yourself.

The starter pool is Fabric-managed and needs no setup. Its nodes are medium and already running, so a session typically starts in a few seconds. You cannot change its node size or family; an admin can only adjust its maximum number of nodes and executors. A custom pool is one you define: node family (memory optimized), node size (from small through extra-large tiers), autoscale with a minimum and maximum node count, and dynamic executor allocation. You can also create a single-node custom pool, where the driver and executor share one node — handy for small workloads. Custom pool nodes are not pre-warmed, so a session takes about three minutes to start.

Autoscale and dynamic allocation are different knobs: autoscale grows or shrinks the pool's node count within its min/max, while dynamic allocation lets each Spark job request more or fewer executors as its stages need them. Executor count and node size solve different problems: more executors add parallelism, while memory per executor comes from the node size.

A pool holds compute only. If you come from Azure Synapse, unlearn the habit of attaching libraries or a Spark version to a pool: in Fabric, libraries, the runtime version and Spark properties live in an environment (next section), and session timeout lives on the Jobs tab. Starter-pool speed also depends on what the session must install: an environment with custom libraries makes even a starter-pool session take minutes.

Creating workspace pools depends on the capacity, not on workspace roles: while a capacity admin has Customized workspace pools turned off, the workspace can use only the starter pool and capacity-level pools.

An environment is a Fabric item that packages everything a Spark session needs beyond raw compute: the Spark runtime version, public libraries (for example from PyPI or Conda), custom libraries you upload (.whl, .jar, .tar.gz), Spark properties and resources — and, if allowed, compute settings such as driver and executor cores and memory. You install libraries once in an environment instead of running pip install in every notebook.

The Environment tab of the workspace Spark settings sets two things: the workspace runtime version and, optionally, a workspace default environment. Items that do not attach their own environment inherit the default; items that attach one use it. When no default environment is set, sessions use the runtime selected from the workspace runtime list. To pin every item in one workspace to a runtime, set it there (or on the default environment) — not in a notebook. %%configure sets session properties such as memory and cores for one session; it cannot change the runtime.

Environment changes take effect only after you publish, and only for sessions started afterwards. Publishing has two modes: in Quick mode libraries are installed as each session starts; in Full mode Fabric prepares a snapshot that is deployed at session start. Either way, custom libraries add start-up time, so a starter-pool session with them takes minutes rather than seconds. Attaching an environment that has no custom libraries or Spark properties keeps the fast start.

Who may customize compute: two near-twin toggles

An environment's compute settings take effect only while the workspace item toggle is on; the capacity toggle governs creating workspace pools, not item-level overrides. The same item toggle also governs Airflow: when it is off, Airflow jobs run on the starter Airflow pool regardless of the workspace's custom Airflow pool.

High concurrency mode lets several notebooks share one Spark session instead of each starting its own, cutting session-start waits and capacity use. It is on by default in every workspace (the workspace admin can turn it off on the High concurrency tab). A user starts a high concurrency session from one notebook and attaches others to it; each notebook keeps its own REPL context, so variables do not collide.

Notebooks share a session only when all of these match:

• the same user runs them;
• the same default lakehouse;
• the same Spark compute configuration;
• the same library packages (the same environment).

Inline installs such as %pip install inside a notebook do not block sharing — they apply to that notebook's context. By default a session holds up to five notebooks; setting the Spark property spark.highConcurrency.max in an environment raises the limit (up to 50).

Pipelines. A separate workspace option, For pipeline running multiple notebooks, extends sharing to pipeline notebook activities. Give the notebook activities the same session tag; activities with a matching tag share a session only if the conditions above also match, so notebooks with a different default lakehouse still get their own session. With the five-per-session default, a sixth notebook with the same tag starts a new session.

The trigger for this feature is a scenario about startup waits or idle sessions across many notebooks. Bigger nodes or more executors raise cost without addressing start-up overhead.

The Jobs tab of the workspace Spark settings controls how long sessions live, how jobs are admitted against capacity cores, and when runaway jobs are cancelled.

• Session timeout ends an idle interactive session after a set time — 20 minutes by default. The timeout applies to idle interactive sessions and is set here, not on the pool.
• Job admission. By default Fabric uses optimistic job admission: a job is admitted when its minimum cores are available and scales up toward its maximum only while capacity cores are free. When no cores are free, interactive jobs get a capacity error and batch jobs queue. Turning on Reserve maximum cores for active Spark jobs admits each job only if its maximum cores are available and holds them, so fewer jobs run at once.
• Set maximum job lifetime cancels user jobs that run past a limit you set. It applies to user jobs however they start — interactive or scheduled notebooks, notebooks run by a pipeline, Spark job definitions, Livy and high concurrency sessions. System-managed jobs such as lakehouse table maintenance (optimize, vacuum) and materialized lake view refreshes are exempt.

Reserved admission trades concurrency for guaranteed scale-up; lowering the pool maximum trades scale-up for concurrency. Optimistic admission is the default because it gives the most concurrent jobs without capping any of them.

A domain is a tenant-level grouping of workspaces by business area — Finance, Sales, Supply Chain — that supports a data-mesh operating model and lets people filter the OneLake catalog by business area. A domain organizes and governs; it does not change who can see or access a workspace, its compute, or its workspace roles. A workspace belongs to at most one domain or subdomain.

A domain contributor must also be workspace admin of the workspace they assign — a Member role is not enough. Subdomains have no admins of their own; they are managed by the parent domain's admins.

Ways to assign workspaces.

• Default domain (set by a Fabric admin or a domain admin): list users or security groups. Workspaces those people administer that have no domain — existing ones and ones created later — join the domain. It never overrides a workspace that already has a domain.
• Bulk assignment by workspace name, by workspace admin, or by capacity: acts only on workspaces that exist now. It can move a workspace out of its current domain only when the tenant setting allowing tenant and domain admins to override workspace assignments is on.
• Workspace settings: a workspace admin who is a domain contributor (or a domain/Fabric admin) picks the domain.

Delegated settings. Some tenant settings can be delegated so a domain admin sets them for their domain — notably certification (who may certify items) and the default sensitivity label.

OneLake is the single, tenant-wide data lake behind every Fabric workspace: each workspace is a container in it, and items store their data under that container. The workspace OneLake settings tab has two controls — shortcut caching and OneLake diagnostics — and a tenant setting governs access from outside Fabric.

Shortcut caching

Shortcut caching stores files read through external shortcuts in OneLake, so repeated reads avoid the source's egress charges and latency. It covers only Google Cloud Storage, Amazon S3, S3-compatible and on-premises (gateway) shortcuts; ADLS Gen2, Azure Blob and internal OneLake shortcuts are not cached. You set a retention period of 1 to 28 days; each access resets a file's retention clock, so a file read more often than the period stays cached. Files larger than 1 GB are never cached and always come from the source. If the source holds a newer version of a cached file, OneLake serves the source version and refreshes the cache, so caching does not serve stale data. Reset cache clears it.

OneLake diagnostics

OneLake diagnostics records data access events — who or what read, wrote or deleted files, including access by external tools through the ADLS APIs — as files in a lakehouse you choose, where you can query them. Rules to know:

• The destination lakehouse must be on the same capacity as the workspace being logged.
• The person enabling it needs workspace admin on the logged workspace and at least Contributor on the destination lakehouse's workspace.
• Events can take up to about an hour to start flowing.
• Moving the workspace to another capacity disables diagnostics; select a destination on the new capacity to resume.
• You can make the diagnostic logs immutable for a retention period, so they cannot be changed or deleted.

Reaching OneLake from external tools

OneLake exposes an ADLS Gen2-compatible endpoint, so tools that speak the ADLS API or ABFS driver can read it in place — if the tenant setting Users can access data stored in OneLake with apps external to Fabric is on. In the path the workspace is the container and the item carries its type suffix: https://onelake.dfs.fabric.microsoft.com/<workspace>/<item>.<itemtype>/Tables/<table> or …/Files/… — for example /Finance/LedgerLH.Lakehouse/Tables/journal. Because the workspace name is part of every path, treat it as a stable identifier.

Not a OneLake workspace setting: how long a deleted workspace is kept before permanent removal is a tenant-wide workspace retention setting in the admin portal. During that window an admin can restore the workspace with its items and data.

An Apache Airflow job is Fabric's managed Airflow: a workspace item that runs Python-defined DAGs without you running or patching any Airflow infrastructure. Teams with existing DAGs, or that need code-first orchestration across systems, keep the Airflow programming model inside Fabric, billed against the capacity. A Fabric data pipeline is the low-code alternative: choose it for drag-and-drop orchestration of copy and transformation activities, and an Airflow job when the orchestration should be Python code or already exists as DAGs. A tenant setting controls whether users can create Airflow jobs.

The workspace Apache Airflow runtime settings choose the default pool for Airflow jobs:

If the workspace admin has turned off Customize compute configuration for items, Airflow jobs use the starter pool even when a custom pool is selected as the default.

Python packages. Add the packages a DAG needs in the Airflow job itself: Settings > Environment configuration > Apache Airflow requirements. Fabric environment items serve Spark sessions only, and a notebook's installs do not reach the Airflow runtime. Inside the job you author or upload DAG files, or sync them from a Git repository, and operators can trigger Fabric items such as pipelines and notebooks.

Put the pieces together. A retail company creates a sales-analytics workspace. Requirements: notebooks must start fast during development; a nightly job needs larger memory-optimized nodes; every notebook needs the same forecasting library; runaway jobs must not run all night; the workspace must sit under the Sales business area; and orchestration will reuse migrated Airflow DAGs.

Fast development sessions → keep the starter pool as the default, and keep heavy custom libraries out of the environment development notebooks use. The nightly job → confirm Customized workspace pools is on for the capacity, create a custom pool with larger nodes and an autoscale cap, and keep Customize compute configuration for items on so the job's environment can select it. Shared library → put it in an environment, publish it, and set it as the workspace default. Runaway jobs → Set maximum job lifetime on the Jobs tab, knowing table maintenance and materialized lake view refreshes are exempt. Business area → the workspace admin, as a domain contributor of Sales, assigns it in workspace settings; if no Sales domain exists, a Fabric admin must create it. Migrated DAGs → create an Apache Airflow job, add its packages under Apache Airflow requirements, and choose an always-on custom Airflow pool for production.

Every requirement resolved to a setting at a specific level, on a specific tab. That mapping is the skill this objective examines.