Lakeflow Jobs Tasks, Dependencies and Control Flow: Retries, Run If, If/Else and For Each (Databricks Data Engineer Associate)

A Lakeflow Jobs job (formerly Databricks Workflows jobs) is a set of tasks plus the dependencies between them, and Databricks runs those tasks in the order the dependencies allow. The dependencies form a directed acyclic graph (DAG): each task lists the tasks it Depends on, and it can start only when those upstream tasks have finished and its Run if condition is met. A task with no dependencies starts as soon as the run starts.

Two shapes cover most designs:

• Fan-out: independent tasks with no dependency on each other start in parallel (compute permitting). Four extracts from separate source systems that each take about fifteen minutes finish in roughly fifteen minutes together, not an hour, when none of them depends on another.
• Fan-in: a task that needs several inputs lists every prerequisite in Depends on, so it waits for all of them. Depending on only the slowest load is a bug: the others may not be finished when it starts.

The rule that keeps a DAG fast is depend on what the task needs, nothing more. A report that needs only the orders and customers tables should depend on those two loads, not on a cleanup task that runs after all loads; the scheduler then starts it the moment its own inputs are ready. Chaining unrelated tasks, or splitting the work into separate jobs glued together with fixed time offsets, throws away both the parallelism and the dependency check.

Each run of the job records a status for every task and an overall status for the run, and failed runs can be repaired instead of rerun from scratch. The rest of this lesson covers the task types the exam names, the control-flow features (retries, Run if, If/else, For each), how values move between tasks, and how run status and repair work. Schedules and trigger types (scheduled, file arrival, table update) are covered in the sibling lesson on schedules and triggers.

The task type decides what a step runs and on which compute. The exam guide names notebook, SQL query, dashboard and pipeline tasks; the Run Job task is the tool for reusing whole jobs.

Other task types exist (Python script, Python wheel, JAR, dbt, and the control-flow tasks If/else and For each), but the four named in the guide plus Run Job carry most of the scenarios.

Notebook tasks: source, parameters and compute

A notebook task runs one notebook as a step of the job. Three settings matter:

• Source. Workspace runs the notebook at a workspace path (including a Git folder, which is a workspace copy someone must pull). Git provider runs the notebook straight from a remote repository at a branch, tag or commit, so every run uses committed code. A job has one Git source: all its Git-sourced tasks share one repository. To orchestrate notebooks that live in two repositories, give each repository its own job and chain them with a Run Job task.
• Parameters. Task parameters are key-value pairs that reach the notebook as widget values, so the notebook reads them with dbutils.widgets.get("source_path"). They are not Spark configuration, environment variables or task values. Reading parameters through widgets keeps one notebook reusable across jobs that pass different values.
• Compute. A notebook task can run on job compute (a cluster created for the run and terminated after it), serverless compute, or an existing all-purpose cluster. It can run on a SQL warehouse only when the notebook is a SQL notebook, written in SQL with SQL as its default language. A Python notebook full of %sql cells still does not qualify; granting permissions or switching warehouse type does not change that.

Note the difference between %run and a Run Job task: %run inlines another notebook's code into the caller's session, while a Run Job task triggers a separate job with its own tasks and settings.

A SQL task runs a Databricks SQL object on a serverless or pro SQL warehouse; a classic warehouse or a job cluster cannot run it. Its subtypes are Query, Alert and File (dashboard refreshes are not a SQL task subtype; they use the Dashboard task below):

• Query runs a saved query, for example to rebuild a gold summary table after an upstream notebook.
• Alert evaluates a saved Databricks SQL alert (a query plus a threshold condition) and notifies its subscribers. It reuses the threshold logic analysts already maintain instead of re-implementing it in a notebook that raises an error.
• File runs a .sql file, which can hold several statements separated by semicolons, from the workspace or from the job's Git source. For a Git source the path is relative to the repository root with no leading / or ./, for example ingest/sql/load_customers.sql; a /Workspace/... path points at the workspace, not the repository.

A Dashboard task refreshes a published AI/BI dashboard and can email the result to subscribers with a PDF snapshot attached. It also needs a serverless or pro SQL warehouse. A draft dashboard cannot be selected: the author must publish it first, and the user configuring the task needs at least CAN VIEW on the published dashboard. When the job runs, the refresh runs as the job's Run as identity. Rerunning the dashboard's queries in a SQL query task does not refresh the published dashboard or send it to anyone.

A Pipeline task starts one update of an existing Lakeflow Spark Declarative Pipelines pipeline and finishes when that update completes, so downstream tasks can depend on fresh streaming tables and materialized views. Running the pipeline's source notebook as a plain notebook task does not update the pipeline's tables; declarative code is executed by a pipeline update.

• Full refresh. Normally an update processes only new data. Enabling full refresh on the Pipeline task makes the update recompute the pipeline's tables from scratch, the right tool for a one-off rebuild after historical source data was corrected. Retries, repairs or a continuous job do not reprocess data already handled.
• One update at a time. A pipeline runs only one update at a time, so a job that contains a Pipeline task runs one job run at a time even if its maximum concurrent runs is set higher. Parallel backfills through the same pipeline queue behind each other.

Run Job tasks: modular orchestration with child jobs

A Run Job task triggers another job in the workspace as one step of the current job and waits for that child run to finish. It is how a team reuses a tested job (an ingestion job, say) inside several orchestrations while keeping its tasks, settings and fixes in one place.

• Parameters. The Run Job task can set values for the child job's job parameters, overriding the child's defaults for that run. Two Run Job tasks pointing at the same child job with different values (for example region = emea and apac) run the same logic twice, in parallel if the child job allows enough concurrent runs.
• Isolation. The child job is a separate run. Task values set in the parent run are not visible inside the child run; pass data to it through its job parameters.
• Crossing Git repositories. Because each job has one Git source, chaining a job per repository with Run Job tasks is how one orchestration runs code from several repositories.
• Multi-step loop bodies. A For each task has exactly one nested task. When each iteration needs several steps (validate, then load), make the nested task a Run Job task whose child job contains those steps.

A retry policy on a task makes Databricks rerun that task automatically when it fails, without rerunning the rest of the job and without code changes. It is the answer to transient failures such as a briefly unavailable API. The policy has:

• Retries: how many extra attempts follow the first one. Two retries means up to three attempts.
• Wait between retries: the minimum interval before the next attempt, measured from the start of the failed attempt, giving a transient error time to clear.
• Retry on timeout: off by default. A task's timeout applies to each attempt; an attempt that exceeds it ends as Timed out, and it is retried only when retry on timeout is enabled. Raising the timeout or the wait does not make a hung attempt retry.

Worst-case duration follows from those rules: with retry on timeout enabled, a task can occupy roughly (retries + 1) × timeout when every attempt hangs, because each attempt gets the full timeout and an interval shorter than an attempt has already elapsed when the attempt times out. A job-level timeout, by contrast, limits the whole run and does not cause any retry.

Related settings are often confused with retries: a duration warning threshold only notifies when a task or run is slow, and maximum concurrent runs controls how many runs of the job may overlap. Neither reruns a failed task.

Run if is the dependency condition on a task: it decides, from the results of the task's direct dependencies, whether the task runs. The default is All succeeded. In the job definition the values appear as run_if constants.

When a condition is not met the task does not run, and the status it gets matters for everything downstream:

• Excluded: the task was skipped because an At least one failed or All failed condition was not met (nothing failed, so the handler had nothing to do), or because it sits on the branch an If/else task did not take. Downstream Run if checks treat an Excluded task as successful.
• Upstream failed: the task was skipped because any other condition was not met, typically All succeeded after a dependency failed. Downstream checks treat it as failed, so the failure propagates: a task with None failed that depends on an Upstream failed task does not run either. An Upstream canceled dependency is treated as failed in the same way.

One exception to "Excluded counts as successful": if all of a task's dependencies are Excluded, the task itself is Excluded, whatever its Run if condition. Exclusion cascades, so a task whose only dependency is an Excluded handler does not run.

For example, when every extract succeeds, a page_oncall task set to All failed is Excluded; a refresh_reports task that depends on both page_oncall and a successful build_marts task still runs, because one dependency succeeded and the Excluded one counts as successful. When an extract fails instead, a conform task with All succeeded downstream of it is Upstream failed, and a task with None failed that depends on conform is skipped as well.

The overall status of a job run is decided by its leaf tasks, the tasks no other task depends on:

Two consequences are tested. First, an always-run leaf (an audit or cleanup task with All done) that succeeds turns a run with a failed ingest into Succeeded with failures, so an alerting rule that fires only on Failed runs stays silent; alert on the task, or use task-level notifications, when that matters. Second, a handler that succeeds does not rescue the run if another leaf was skipped: if enrich fails, a downstream export_feed leaf with All succeeded is Upstream failed and the run is Failed, even though a log_incident handler with At least one failed ran and succeeded. Status never depends on which task finished last. Task-level statuses such as Upstream failed and Excluded describe individual tasks, not the run.

Repairing failed runs

Repair run is for unsuccessful runs (failed or canceled): it finishes such a run by rerunning only the unsuccessful tasks and their dependents (failed, timed out, canceled, and the skipped tasks downstream of them). Tasks that already succeeded are not rerun, which matters when they are not idempotent, for example appends. The repair is recorded as part of the same run.

• Current settings. Repaired tasks run with the job's and tasks' current settings, so fixing a cluster's memory or a notebook and then repairing picks up the fix.
• Parameter overrides. The repair dialog accepts new parameter values that apply to the repaired tasks, so a wrong run_date can be corrected without editing the saved job definition.

Compare the alternatives: Run now with different parameters starts a brand-new run of every task, duplicating appends; retries repeat a task with the same inputs; and repair applies only to unsuccessful runs, so it is not a way to reprocess data that already loaded successfully.

The If/else condition task branches a job: it evaluates one comparison and runs the tasks attached to its true outcome or its false outcome. Downstream tasks pick an outcome when they declare the condition task as a dependency; tasks on the branch not taken are Excluded. Branching this way lets a job route a batch (quarantine versus publish) without any task failing, which is cleaner than raising an exception and catching it with an At least one failed condition.

The condition is left operand, operator, right operand. Operands can be literals, job parameters such as {{job.parameters.mode}}, task values such as {{tasks.profile_batch.values.null_pct}}, or other dynamic value references.

Run if conditions cannot do this job: they see only whether upstream tasks succeeded or failed, never a value a task produced. And job parameters cannot carry a value computed mid-run, because they are fixed when the run is triggered. To branch on something computed during the run, set it as a task value and compare it in an If/else task.

The For each task runs one nested task once per element of an input array, and tracks each iteration separately, so a failure for one value is visible and does not hide the others. It replaces defining dozens of near-identical tasks, or a single notebook that loops internally (where every value shares one task run).

• Referencing the element. The nested task reads the current element with {{input}}, typically passed as a task parameter. When the inputs are JSON objects, {{input.<key>}} reads one field, for example {{input.region}} and {{input.table}}.
• One nested task, never another For each. Nested loops are not allowed, so loop over combinations by flattening them into one array of objects. When each iteration needs several steps, the nested task can be a Run Job task that runs a child job.
• Concurrency. Iterations run one at a time by default (concurrency 1). Raising the For each task's concurrency runs several iterations in parallel. The job's maximum concurrent runs setting is unrelated: it governs overlapping runs of the whole job, not iterations inside one run, and adding workers does not help a loop that starts iterations one by one.

The inputs can come from three places, chosen by when the list is known:

Values reach tasks in three ways, and the exam tests which one fits a requirement.

Job parameters suit one value every task needs, chosen per run, such as env = dev or prod: define it once instead of copying a task parameter onto every task. A notebook reads it with dbutils.widgets.get. When a job parameter and a task parameter share a key, the job parameter takes precedence, whether the run is manual or scheduled. Job parameters cannot change during a run.

Task values carry small results from one task to later tasks in the same run, regardless of the compute each task uses. Spark configuration, temporary views and widgets do not cross task boundaries, because each task runs in its own session.

# upstream task named stage_orders
dbutils.jobs.taskValues.set(key="batch_id", value=batch_id)

# downstream notebook
batch_id = dbutils.jobs.taskValues.get(
    taskKey="stage_orders", key="batch_id",
    debugValue="test-batch")

Downstream tasks can also receive a task value as a parameter with the reference {{tasks.<task_name>.values.<value_name>}}; the task name comes first. debugValue is returned only when the notebook runs outside a job, which makes interactive testing possible while scheduled runs still read the real value. default is returned inside a job when the key is missing, which can hide a broken upstream task, so Databricks does not recommend relying on it. A task value is stored as JSON and its JSON representation cannot exceed 48 KiB; pass larger data by reference, writing it to a table or volume file and passing the location as the task value.