Fabric Orchestration: Dataflow Gen2 vs Pipeline vs Notebook, and Triggers

Choose by asking three questions: who builds it, how complex is the transformation, and does it need to coordinate other work? Dataflow Gen2 is for low-code transformation — Power Query Online with a large connector library, built by analysts or engineers who prefer a visual editor, landing results in a Lakehouse, Warehouse, Eventhouse KQL database, or other output destination. A data pipeline is for orchestration and large-scale data movement — the Copy activity plus control-flow activities that schedule, sequence, branch, and retry other workloads; it transforms very little by itself. A notebook is for code-first work — PySpark, Spark SQL, Scala, or R against Lakehouse tables, where transformations are too complex, too dynamic, or too performance-sensitive for a visual tool.

The tools compose rather than compete: a common production design is a pipeline that copies raw files into a Lakehouse, invokes a notebook to transform bronze to silver to gold, then refreshes a semantic model — with a Dataflow Gen2 handling a side-channel of business-owned reference data. When an exam scenario stresses "minimal code" or "business analysts," answer Dataflow Gen2; "orchestrate," "schedule," or "copy large volumes," answer pipeline; "complex logic," "custom code," or "Spark," answer notebook.

Dataflow Gen2 is Power Query running as a Fabric item: you connect to a source, shape the data through applied steps (filter, merge, pivot, custom columns in M), and write the result to an output destination such as a Lakehouse table, a Warehouse table, an Eventhouse KQL database, or an Azure SQL database. Because the editor shows data previews at every step, it is the fastest path for connector-heavy, moderately complex preparation work — and the only one of the three tools a non-programmer can own end to end.

Its orchestration story is deliberately thin: a dataflow can be refreshed on a schedule or invoked from a pipeline with the Dataflow activity, but it cannot itself invoke notebooks, branch on conditions, or loop. That is the boundary the exam probes. A dataflow also supports incremental refresh patterns to avoid reprocessing unchanged data on every run.

Choose Dataflow Gen2 when the source needs one of its many connectors, the transformation fits Power Query comfortably, and the owner prefers visual tooling. Avoid it when transformations demand code-level control or when the task is really orchestration wearing a transformation costume — sequencing steps, conditional retries, and fan-out belong in a pipeline.

A data pipeline is Fabric's orchestrator: a canvas of activities connected by success, failure, completion, and skip dependencies. The Copy activity moves data at scale between a wide range of sources and destinations with light schema mapping. Around it sit invocation activities — Notebook, Dataflow, Stored procedure, Script, Semantic model refresh, Invoke pipeline — and control-flow activities: ForEach (iterate a collection), If Condition and Switch (branch), Until (loop to a condition), Wait, Lookup (fetch rows or config to drive logic), Get Metadata (inspect files and folders), and Set Variable.

Dependency conditions are the error-handling primitive: route the red "on failure" path to a Teams or Outlook activity for alerting, or use "on completion" when a step must run regardless of outcome. Activities also carry retry counts and timeouts, which is why pipelines — not notebooks calling notebooks — are the right home for production reliability requirements.

The pipeline's own transformation vocabulary is intentionally small. If you find yourself contorting Copy activities to reshape data, the design smell is real: land the data, then hand the reshaping to a notebook or dataflow. The pipeline's job is when, in what order, and what happens on failure — not how the bytes are transformed.

A Fabric notebook runs code on Spark against OneLake data — typically PySpark or Spark SQL reading and writing Lakehouse Delta tables. It is the tool for transformations that need joins across many tables, window functions, data quality rules, machine-learning steps, or any logic you want unit-testable and version-controlled. Notebooks attach to a default Lakehouse, whose tables and files are directly addressable, and can reach other lakehouses through absolute paths or shortcuts.

Notebooks participate in orchestration three ways. First, a pipeline's Notebook activity runs them with parameters — the standard production pattern. Second, notebooks can be scheduled directly from their own settings when a single notebook is the whole job. Third, notebooks can chain each other with notebookutils: notebookutils.notebook.run() executes a child notebook and returns its exit value, and notebookutils.notebook.runMultiple() executes a set of notebooks — optionally as a dependency graph — inside the same Spark session, avoiding the session-startup cost of separate pipeline activities.

That last point is a genuine design trade-off the exam can probe: orchestrating ten small notebooks as ten pipeline Notebook activities gives per-activity monitoring and retries but may pay repeated session overhead; runMultiple inside one driver notebook shares a session and runs a DAG efficiently, at the cost of coarser monitoring. Reach for pipelines when steps are heterogeneous (copy + notebook + procedure) or need individual retry policies; reach for runMultiple when many related Spark steps should share compute.

Fabric starts work two ways: on the clock or on an event. Schedules are configured per item — pipelines, notebooks, and dataflows each have scheduled-run settings where you define frequency (by the minute, hourly, daily, weekly), start and end dates, and time zone. A schedule fires the item regardless of what data has or has not arrived, which makes it the right choice for batch cadences like "load the warehouse nightly" — and the wrong choice when the real requirement is "process each file as it lands."

Event-based triggers answer that second requirement. Fabric's eventing runs through the Real-Time hub and Activator (the evolution of Data Activator/Reflex): a pipeline trigger subscribes to events — OneLake events (files created or modified in a lakehouse) or Azure Blob Storage events (blob created in an external storage account), among other event sources — and starts the pipeline when a matching event arrives. You can filter the subscription (by container, folder path, or file name pattern) so only relevant arrivals fire the trigger, and the event's metadata — such as the arriving file's name and folder path — is exposed to the pipeline as built-in trigger parameters you reference in dynamic expressions, letting one parameterized pipeline process whichever file arrived.

Activator itself generalizes beyond pipeline triggers: it watches streaming data and event sources for conditions (a threshold crossed, a pattern detected) and takes actions such as starting a Fabric job, sending an alert, or calling Power Automate. For DP-700, the essential mapping is: fixed cadence → schedule; react to file or data arrival → event-based trigger via Activator/Real-Time hub; react to conditions in streaming data → Activator rules. Event-driven beats frequent polling schedules both on latency and on wasted runs — the classic exam justification.

Parameters make one pipeline serve many workloads. A pipeline parameter is declared on the pipeline with a type and optional default, is read-only during the run, and is referenced in any activity setting through dynamic content: @pipeline().parameters.SourceFolder. Callers supply values — a manual run prompts for them, a schedule can fix them, an Invoke Pipeline activity maps them, and an event trigger feeds them from event metadata. Variables, by contrast, are mutable during the run: Set Variable writes them, @variables('rowCount') reads them — use variables for state that changes mid-run, parameters for run configuration.

The expression language glues everything together. Expressions start with @ and compose functions: @concat('sales/', pipeline().parameters.Region, '/data.parquet') builds paths; @formatDateTime(utcnow(), 'yyyy/MM/dd') builds date partitions; @activity('LookupConfig').output reads a previous activity's output — the pattern that lets a Lookup activity's result drive a ForEach: set the ForEach items to @activity('LookupConfig').output.value and reference the current element inside the loop with @item() (for example @item().tableName).

System functions like @pipeline().RunId, @pipeline().TriggerTime, and trigger-event metadata round out the vocabulary. The exam pattern to internalize: any answer that hardcodes a file name, date, or table list in ten near-identical pipelines is wrong when a single parameterized pipeline with dynamic expressions can do the job.

Values flow into a notebook through the Notebook activity's base parameters. In the notebook, you designate one cell as the parameters cell (toggle it in the cell options) and declare default values there — region = 'emea', load_date = ''. At run time, Fabric injects the base parameters as a new cell that overrides those defaults, so the notebook runs interactively with defaults and runs parameterized under orchestration with no code changes. Match the parameter names and types exactly; a mismatched name silently leaves the default in place.

Values flow back through the notebook's exit value: end the notebook with notebookutils.notebook.exit('42137') and the calling pipeline reads it at @activity('TransformNotebook').output.result.exitValue — feed it to an If Condition to branch on row counts or status codes, or pass it as a base parameter to the next notebook. Exit values are strings, so serialize anything structured (JSON works well) and parse it on the other side.

Session configuration is parameterizable too: the %%configure magic command in a notebook's first cell can set Spark session options and — importantly — the default Lakehouse binding, and it supports parameterization from a pipeline so the same notebook can run against dev and prod lakehouses. Together, base parameters (data in), exit values (results out), and parameterized configuration (environment binding) are the complete notebook-pipeline contract DP-700 expects you to know.

Three patterns cover most production designs. Parent-child pipelines: a parent pipeline sequences stages and calls child pipelines with the Invoke Pipeline activity, passing parameters down — each child stays small, testable, and independently runnable. Metadata-driven ingestion: a Lookup activity reads a control table listing sources (table name, watermark column, destination), a ForEach iterates it, and inside the loop a parameterized Copy activity and Notebook activity process @item() — onboarding a new source becomes a row insert, not a new pipeline. Notebook DAGs: a driver notebook uses notebookutils.notebook.runMultiple() to run related Spark transformations with dependencies inside one shared session.

Scenario: a retailer's suppliers drop CSV files into an Azure Blob Storage container at unpredictable times; each file must be ingested into a bronze Lakehouse table, transformed to silver with data-quality rules, and the semantic model refreshed — with a Teams alert on any failure. The design: an event-based trigger subscribes to blob-created events on the container, filtered to .csv, and starts one parameterized pipeline, passing the file name and folder path from the trigger's event metadata into pipeline parameters. A Copy activity uses those parameters in dynamic expressions to land exactly that file in the bronze area. A Notebook activity passes the file name as a base parameter; the notebook validates and merges to silver, then exits with the accepted row count via notebookutils.notebook.exit. An If Condition on @activity('Transform').output.result.exitValue routes zero-row loads to a warning path; the success path runs a Semantic model refresh activity; every activity's failure path converges on a Teams activity.

Notice what the scenario did not use: no polling schedule (events beat polling on latency and cost), no logic in the pipeline that belonged in the notebook, and no per-supplier pipeline copies — one parameterized pipeline handles every file. That reasoning, applied under exam time pressure, is the skill this topic measures.