Fabric Orchestration: Dataflow Gen2 vs Pipeline vs Notebook, and Triggers

Choose by asking four questions: who builds it, how complex is the transformation, how much data moves, and does the work need to coordinate other steps? Dataflow Gen2 is for low-code transformation — Power Query Online with a large connector library, built by analysts or engineers who prefer a visual editor, landing results in a Lakehouse, Warehouse, Eventhouse KQL database, or other output destination. A data pipeline is for orchestration and large-scale data movement — the Copy activity plus control-flow activities that schedule, sequence, branch, and retry other workloads; it transforms very little by itself. A notebook is for code-first work — PySpark, Spark SQL, Scala, or R against Lakehouse tables, where transformations are too complex, too dynamic, or too performance-sensitive for a visual tool.

Two facts often decide the choice. First, notebooks cannot use the on-premises data gateway, so a source reachable only through a gateway must be read by a gateway-capable tool — for high-volume, transformation-light extraction that is the pipeline Copy activity, which is built for bulk throughput (Dataflow Gen2 also uses the gateway but routes everything through Power Query, which adds cost and time when nothing is being transformed). Second, connectivity and output format rarely separate Dataflow Gen2 from a notebook: Dataflow Gen2 can call web and REST APIs (the Web connector, Web.Contents) and write Delta tables to a lakehouse, just as a notebook can. What separates them is the logic — paging through continuation tokens, per-record enrichment, reuse of a team's tested Python library — and the skills of the people who own it.

The tools compose rather than compete. When no single tool satisfies every constraint, split the work: a Copy activity lands gateway-only data in the lakehouse, then a notebook applies the Python logic, all sequenced by one pipeline. A common production design is a pipeline that copies raw files into a Lakehouse, invokes a notebook to transform bronze to silver to gold, then refreshes a semantic model — with a Dataflow Gen2 handling business-owned reference data.

Two neighbours are worth placing. Fabric user data functions are lightweight Python functions hosted in Fabric and called on demand; they are useful for small pieces of reusable logic but are not a distributed engine, so large-scale transformations over billions of rows belong in Spark notebooks. A warehouse stored procedure handles set-based T-SQL at scale well, but cannot reuse Python code.

Dataflow Gen2 is Power Query running as a Fabric item: you connect to a source, shape the data through applied steps (filter, merge, pivot, rename, change types, custom columns in M), and write the result to an output destination such as a Lakehouse table, a Warehouse table, an Eventhouse KQL database, or an Azure SQL database. Because the editor shows data previews at every step, it is the fastest path for connector-heavy, moderately complex preparation work — and the only one of the three tools a non-programmer can own end to end.

Its orchestration story is deliberately thin: a dataflow can be refreshed on its own schedule or invoked from a pipeline with the Dataflow activity, but it cannot itself invoke notebooks, branch on conditions, or loop. A dataflow also supports incremental refresh patterns to avoid reprocessing unchanged data on every run.

Choose Dataflow Gen2 when the source needs one of its many connectors, the transformation fits Power Query comfortably, and the owner prefers visual tooling. Avoid it when transformations demand code-level control (custom iteration, reuse of Python code, unit tests in a repository), when the job is bulk movement with no transformation, or when the task is really orchestration — sequencing steps, conditional retries, and fan-out belong in a pipeline.

A data pipeline is Fabric's orchestrator: a canvas of activities connected by dependency conditions. The Copy activity moves data at scale between a wide range of sources and destinations, including on-premises sources through the data gateway, with light schema mapping. Around it sit invocation activities — Notebook, Spark job definition, Dataflow, Stored procedure, Script, Semantic model refresh, Invoke pipeline — notification activities (Teams, Office 365 Outlook), and control-flow activities: ForEach (iterate a collection), If Condition and Switch (branch), Until (loop to a condition), Wait, Lookup (fetch rows or config to drive logic), Get Metadata (inspect files and folders), Set Variable, Append Variable and Fail.

Each activity has general settings for timeout and retry count and interval, which is why pipelines — not notebooks calling notebooks — are the natural home for production reliability requirements. Dependency conditions (Upon success, Upon failure, Upon completion, Upon skip) decide what runs next; the section on error handling below explains how they combine.

The pipeline's own transformation vocabulary is intentionally small. If you find yourself contorting Copy activities to reshape data, land the data, then hand the reshaping to a notebook or dataflow. The pipeline's job is when, in what order, and what happens on failure — not how the bytes are transformed.

A Fabric notebook runs code on Spark against OneLake data — typically PySpark or Spark SQL reading and writing Lakehouse Delta tables. It is the tool for transformations that need joins across many tables, window functions, deduplication with custom rules, data quality checks, machine-learning steps, or any logic you want unit-testable and version-controlled. Notebooks attach to a default Lakehouse, whose tables and files are directly addressable by relative paths, and can reach other lakehouses through absolute paths or shortcuts.

Notebooks participate in orchestration three ways: a pipeline's Notebook activity runs them with parameters (the standard production pattern); a notebook can be scheduled directly from its own settings when it is the whole job; and notebooks can chain each other with notebookutils.

There are therefore three ways to run many notebooks, each with a different trade-off. Separate pipeline Notebook activities give per-activity monitoring, retries and timeouts, but each activity normally starts its own Spark session. High concurrency mode for pipelines — enabled in the workspace's Spark settings — lets Notebook activities that carry the same session tag share one session. Sharing also requires the same user, the same default lakehouse, the same Spark compute configuration and the same library packages; a shared session holds at most five notebooks, and once it is full the next activity starts another high-concurrency session rather than failing. runMultiple() in a driver notebook runs a whole DAG — dependencies plus parallel branches — in one session with no five-notebook ceiling. Its concurrency is still bounded: the DAG's concurrency setting caps how many notebooks run at once (by default three times the available CPU cores; 0 removes the cap), and every parallel notebook shares the one session's compute. Each DAG activity can set its own retry count and per-cell timeout, but the pipeline's monitoring sees only the driver notebook as one activity. Reach for pipeline activities when steps are heterogeneous (copy + notebook + procedure) or need individual retry policies; reach for runMultiple when many related Spark steps should share compute.

Fabric starts work two ways: on the clock or on an event, and a pipeline can use both at once. Schedules are configured per item — pipelines, notebooks, Spark job definitions and dataflows each have schedule settings. A schedule fires the item regardless of what data has or has not arrived, which makes it right for batch cadences like "load the warehouse nightly" and wrong when the requirement is "process each file as it lands".

• Recurrence types: by the minute, hourly, daily, weekly and monthly. Weekly lets you tick specific weekdays and add several times of day, so a cadence of several fixed times on working days only is a single weekly schedule. Daily recurrence always runs on every day of the week, and minute or hourly intervals run around the clock, so neither can express a weekdays-only cadence.
• Start and end: every schedule has a start date and time, an end date and time, and a time zone.
• Several schedules per pipeline: a pipeline can have multiple schedules (up to 20), and each schedule can pass its own parameter values. Two schedules on one parameterized pipeline — for example one passing Market = NorthAmerica early in the morning and another passing Market = AsiaPacific late in the evening — replace two duplicated pipelines. Schedules supply parameters, not variables.
• Failure notifications: the schedule settings can email chosen users or groups when a scheduled run fails, with no extra activity or item. They do not fire for on-demand (manual) runs.

Event-based triggers answer the second requirement. You add a trigger from the pipeline editor; Fabric saves it as an Activator (Reflex) item in a workspace, which subscribes to an event source through the Real-Time hub and starts the pipeline when a matching event arrives. Event triggers react to pushed events — they do not poll. The supported event sources are:

Match the event source to what actually happens: a file appearing anywhere in OneLake (a lakehouse Files folder included) is a OneLake event; a run finishing is a job event; an item being created or changed is a workspace item event. Narrow the subscription with filters — event type (created only, so deletions do not start runs) and fields such as the Subject, for example "ends with .parquet" so temporary or marker files written alongside the data are ignored.

The event's details reach the pipeline as trigger parameters, read with expressions such as @pipeline()?.TriggerEvent?.FileName and @pipeline()?.TriggerEvent?.FolderPath, so one parameterized pipeline processes whichever file arrived. The ?. operator is null-safe: during a manual or scheduled run the trigger parameters are NULL, and @pipeline().TriggerEvent.FileName would fail, whereas the ?. form returns null and lets the run continue (pair it with a default if the activity needs a value).

Activator rules generalize beyond pipeline triggers: Activator watches an eventstream or other event source for a condition over time — a value crossing a threshold, an average staying high for ten minutes — and acts within seconds by sending a Teams message or email, or starting a pipeline or other Fabric item. A business user can own and adjust a rule without code. That is different from an eventstream filter, which checks each event on its own and cannot express "for ten minutes". The mapping: fixed cadence → schedule; react to a file, job or item event → event trigger; react to a condition in streaming data → Activator rule. Event-driven beats frequent polling schedules on latency and on wasted empty runs.

Parameters make one pipeline serve many workloads. A pipeline parameter is declared on the pipeline with a type and optional default, is fixed once the run starts, and is referenced through dynamic content: @pipeline().parameters.SourceFolder. Callers supply values — a manual run prompts for them, each schedule can pass its own, an Invoke pipeline activity maps them, and an event trigger can feed them from event metadata. Variables, by contrast, are mutable during the run: Set Variable and Append Variable write them, @variables('rowCount') reads them. If a value comes from outside the run, it is a parameter; if the run computes and changes it, it is a variable. No activity can reassign a parameter.

Expressions start with @ and are evaluated while the pipeline runs, which is why they can use activity outputs and the current loop element. There are two ways to build a string:

• A function call: @concat('raw/', item().TableName, '/data.parquet').
• String interpolation: embed expressions inside literal text with @{...}, for example raw/@{item().TableName}/data.parquet. concat() is optional.

Reading activity outputs uses @activity('Name').output, and the shape depends on the activity:

Inside a ForEach, @item() returns the current element, so @item().TableName reads a column of the current row. The Lookup's output.value stays the whole array on every iteration; @item() is the only reference that moves with the loop.

Dates and types are common sources of subtle bugs. utcNow() returns the current UTC timestamp; addDays(utcNow(), -1) moves to the previous day, whereas startOfDay(utcNow()) only truncates today to midnight and stays on today. formatDateTime(..., 'yyyy/MM/dd') formats it — uppercase MM is the month, lowercase mm is minutes, so yyyy/mm/dd silently produces wrong folders. Values also keep their types: a notebook exit value is a string, so convert it with int() or float() before passing it to a numeric function such as greater() — for example @greater(int(activity('CountRows').output.result.exitValue), 100). The expression engine does not coerce strings to numbers, and string functions such as length() measure characters, not the value the string represents.

System values like @pipeline().RunId, @pipeline().TriggerTime and the trigger-event parameters round out the vocabulary. Ten near-identical pipelines that differ only in a file name, date or table list are a design smell when one parameterized pipeline with dynamic expressions can do the job.

Values flow into a notebook through the Notebook activity's base parameters. In the notebook, you designate one cell as the parameters cell (toggle it in the cell options) and declare default values there — region = 'emea', load_date = ''. At run time, Fabric injects the base parameters as a new cell after the parameters cell, overriding those defaults, so the notebook runs interactively with defaults and runs parameterized under orchestration with no code changes. Match the parameter names exactly; a mismatched name leaves the default in place. Pipeline expression syntax such as @variables() means nothing inside Python — the value must be passed as a base parameter. Do not confuse this with a Spark job definition, which receives command-line arguments (read from sys.argv) rather than a parameters cell.

Values flow back through the notebook's exit value: end the notebook with notebookutils.notebook.exit(str(row_count)) and the pipeline reads it at @activity('TransformNotebook').output.result.exitValue. Printing a value does not set it. Exit values are strings — convert with int() before numeric comparisons, and serialize structured results as JSON. Use an exit value, not a raised exception, to report a normal data condition such as "zero rows": an exception marks the activity Failed in monitoring.

Parameterized %%configure binds the Spark session itself per run. A parameters-cell value arrives after the session has started, so it cannot change which lakehouse the notebook's relative paths resolve to. To vary the default lakehouse per run, put %%configure in the notebook and replace the value with a parameter object that names a parameterName and a defaultValue:

%%configure
{
  "defaultLakehouse": {
    "name": { "parameterName": "lakehouseName", "defaultValue": "DevLakehouse" }
  }
}

The Notebook activity passes lakehouseName as a parameter; interactive runs fall back to the default. In a pipeline run, %%configure must be the first code cell — placed later it would require restarting the session, and the run fails. Two limits apply: a notebook's own scheduled runs do not support parameterized session configuration (it works when a pipeline Notebook activity runs the notebook), and %%configure in a notebook called through notebookutils.notebook.run() is ignored, because the child runs in the caller's session. Variable libraries solve a different problem — values that differ per deployment stage (development, test, production), not per run. Base parameters (data in), exit values (results out), and parameterized configuration (environment binding) together are the full notebook-pipeline contract.

A ForEach runs its inner activities once per element of an array, and its own settings — not the settings of the activities inside it — decide whether iterations run one at a time or together.

If each table copies quickly but the tables run one after another, the bottleneck is the loop: clear Sequential and set a batch count. Raising the copy settings speeds up a single copy, which is not the problem.

Loops cannot nest. A ForEach or Until cannot contain another ForEach or Until. To loop over regions and then over each region's tables, put the inner loop in a child pipeline: the outer ForEach runs an Invoke pipeline activity that passes @item().Region to a child parameter, and the child runs its own Lookup for that region and its own ForEach over output.value.

Variables in loops need care because variables are scoped to the whole pipeline run, not to an iteration:

• Set Variable overwrites a variable's value, and its expression cannot reference the variable being set — so it cannot build up a list.
• Append Variable adds one element to an existing Array variable. It is the activity for collecting results across iterations, such as each failed table's name on a Copy activity's On failure path. It works only on array variables, never on parameters (parameters cannot change during a run).
• In a ForEach with Sequential cleared, iterations that Set the same variable overwrite each other unpredictably. Read per-iteration values from @item() instead of copying them into a variable.

Pipeline error handling is built from dependency conditions on the arrows between activities: Upon success, Upon failure, Upon completion (either outcome) and Upon skip (the upstream activity never ran). Four rules explain how they behave.

• Several incoming dependencies are ANDed. An activity with On failure arrows from Copy, Notebook and Dataflow runs only if all three fail. In a success-linked chain that never happens, because a failure stops the chain.
• After a failure, downstream activities are skipped, not failed. If the notebook fails, the copy before it succeeded and the dataflow after it is skipped.
• The generic error handler therefore hangs off the last activity of the chain with two paths, On failure and On skip, leading to the handler. Any failure ends with the last activity either failing or being skipped, so the handler runs exactly once for any failing step and never on success.
• Try-catch hides failures. When an activity fails and its Upon failure path runs a handler that succeeds, the pipeline reports the run as Succeeded, because its outcome is judged on the activities that end each path. A logging step on the failure path is therefore not enough when operations rely on a Failed status or the schedule's failure email.

The Fail activity fixes that last case: place it after the handler (on the handler's success path) and the run ends as Failed with a custom error message and error code, such as the original activity's error. As a general rule, only an activity that actually fails — a Fail activity, or the original error left unhandled — makes the run Failed; notification activities report a problem without changing the outcome. Retry settings are a separate tool: they re-run the activity that owns them after a transient error and do not change how dependency conditions route the rest of the pipeline.

The Invoke pipeline activity runs another pipeline as a step, which is how you factor shared logic — audit logging, a standard load — into one parameterized child instead of copying the same activities into many pipelines.

• Parameters are the only way in. The parent maps values (literals or expressions such as @item().Region) to the child's parameters. A child cannot read the parent's activity outputs with @activity(); anything it needs must be passed as a parameter, or the child must fetch it itself.
• Wait on completion decides whether the parent waits. Selected, the Invoke pipeline activity runs until the child ends and takes the child's outcome, so a failed child fails the activity and On success dependencies wait for the child to finish. Cleared, the activity succeeds as soon as it has started the child; following activities run in parallel with the child, and a child failure does not fail the parent. Only Wait on completion ties the parent to the child's outcome: the parent's dependency conditions judge the Invoke pipeline activity itself, and without the setting that activity has already succeeded as soon as the child started.
• Two versions exist. The legacy Invoke pipeline activity calls a Fabric pipeline in the same workspace. The newer version uses a connection and can call Fabric pipelines in other workspaces as well as Azure Data Factory and Azure Synapse Analytics pipelines.

Use Invoke pipeline when a caller must pass values and wait for the result. An event trigger on job events can also start one pipeline when another finishes, but it is decoupled: the caller passes no parameters and does not wait.

Three patterns cover most production designs. Parent-child pipelines: a parent sequences stages and calls child pipelines with Invoke pipeline, passing parameters down (see the previous section). Metadata-driven ingestion: a Lookup (with "First row only" cleared) reads a control table listing sources, a ForEach iterates output.value, and inside the loop a parameterized Copy activity and Notebook activity process @item() — onboarding a new source becomes a row insert, not a new pipeline. Lookup reads rows from a table or query; Get Metadata describes files and folders (child items, size, existence). ForEach iterates a known array and can run iterations in parallel; Until repeats until a condition becomes true; Switch picks one branch by value. Notebook DAGs: a driver notebook uses runMultiple() to run related Spark transformations with dependencies inside one shared session.

Scenario: a logistics vendor uploads JSON shipment exports to an Azure Blob Storage container at irregular times; each export must be landed in bronze, cleansed into silver, and the semantic model refreshed — with an email to the operations team on any failure, while the run still shows as Failed. The design: an event trigger on Azure Blob Storage blob-created events, filtered to subjects ending in .json, starts one parameterized pipeline. A Copy activity uses the trigger parameters (read null-safely with ?.) to land exactly that blob in bronze. A Notebook activity passes the blob name as a base parameter; the notebook cleanses and merges to silver, then exits with the number of rejected records. An If Condition that converts that exit value with int() routes exports with too many rejects to a quarantine path; otherwise a Semantic model refresh runs. The last activity's On failure and On skip paths lead to an Outlook activity followed by a Fail activity.

Notice what the scenario did not use: no polling schedule (events beat polling on latency and cost), no logic in the pipeline that belonged in the notebook, no per-supplier pipeline copies, and no handler that quietly turns failures into successes.