Lakeflow Jobs Schedules and Triggers: Scheduled, File Arrival, Table Update and Continuous (Databricks Data Engineer Associate)

A trigger decides when Lakeflow Jobs starts a new run of a job without anyone clicking Run now. A job can be started manually at any time, and it can have one of these automatic triggers (multiple triggers per job are in Beta):

The exam objective asks you to implement these triggers and to choose between time-based and data-driven ones. The rest of this lesson takes each trigger in turn, then the job settings that interact with every trigger (maximum concurrent runs, queueing, pause and resume), the dynamic value references that tell a task why it ran, and finally the decision itself. How tasks inside a run depend on each other, retry and branch is covered in the sibling lesson on tasks, dependencies and control flow.

A scheduled trigger starts runs on a time-based schedule, and it comes in two forms that differ in how much of the timing you control.

The practical consequence: if a job must run at a particular wall-clock time, or downstream teams rely on the run times not changing, use an advanced schedule. Saving a simple schedule at the "right" moment is fragile, because the next edit and save moves the runs again.

Advanced schedules accept Quartz cron expressions, which differ from the familiar Unix cron in two ways: they start with a seconds field, and one of the two day fields must be ? (no specific value).

Reading a few expressions field by field:

• 0 45 4 * * ? runs at 04:45:00 every day: seconds 0, minute 45, hour 4, any day of month, any month, no specific day of week.
• 0 30 7 ? * SAT,SUN runs at 07:30 on Saturdays and Sundays. Day of week carries the value, so day of month is ?.
• 0 0/10 9-16 * * ? runs every 10 minutes from 09:00 to 16:50 every day. An hours range covers every minute inside its last hour, so 9-16 ends with the 16:50 run and never reaches 17:00.

Because every field is positional, read each expression left to right as seconds, minutes, hours, day of month, month, day of week, and check three things: there are six fields (not Unix cron's five), the step or range sits in the field you meant, and exactly one of the two day fields is ?.

Time zones and daylight saving

The time zone of an advanced schedule decides which clock the schedule follows, and the right choice depends on whether the requirement is a fixed interval or a local clock time.

• Fixed interval, every N minutes or hours, all year: use UTC. Databricks notes that an hourly job scheduled in a time zone that observes daylight saving can be skipped or appear delayed by an hour or two around the clock change. UTC has no clock change, so the period is always exact.
• Fixed local time, such as 09:00 New York every weekday: use the local time zone (America/New_York). Any single UTC time is correct for only part of the year, because the offset from UTC changes with daylight saving.

Problems caused by the schedule itself are not fixed by job settings that act on runs: queueing, a higher concurrency limit or task retries cannot create a run the schedule never produced. Change the time zone instead.

A file arrival trigger starts a run when new files arrive in a monitored location: a path in a Unity Catalog volume or an external location. Databricks checks the location about once a minute, so the run starts shortly after a delivery and no run starts when nothing has arrived.

Rules that decide many scenarios:

• Only new files count. Overwriting an existing file with the same name does not trigger a run. A partner that replaces inventory.json in place each night triggers the job only once, when that file name first appeared. Every delivery needs its own file name (for example one that includes a timestamp) to count as an arrival.
• Subdirectories are checked recursively. One trigger on a landing folder sees files in every dated subfolder below it, including subfolders created later, so there is no need to add a trigger per day or per year.
• The trigger does not tell the job which files are new. The only file arrival dynamic value reference is the monitored location. Several files can be batched into one run and others can land while a run is in progress, so filtering by time misses files and reading the whole path duplicates them. The documented pattern is to have the job read the path with Auto Loader using trigger(availableNow=True) and a checkpoint: it loads every file not yet recorded in the checkpoint, then stops, giving exactly-once loading across batched arrivals and retried runs.

Limits without file events

When the external location is not enabled for file events, the trigger has to list the storage itself, and two limits apply:

• The monitored path can hold at most 10,000 files. The limit applies to the path the trigger monitors, including its subdirectories, not to the whole external location, so pointing the trigger at a smaller subfolder (for example an incoming folder rather than a parent that also contains an archive) can bring it under the limit.
• A workspace can have at most 50 jobs with file arrival triggers on locations without file events.

Databricks recommends enabling managed file events on the external location. Triggers on a location with file events have no file-count limit and do not count toward the 50-job limit. Enabling file events requires the owner of the external location or the MANAGE privilege on it. Timing settings do nothing for either limit.

File arrival and table update triggers share two optional advanced settings, and they solve different problems: one caps how often runs start, the other waits for a quiet period before starting.

When both are set, a run starts at the later of the two conditions. Work an example: the previous run completed at time 0, minimum time between triggers is 600 seconds and wait after last change is 120 seconds. Updates arrive at 100 and 550 seconds. Nothing can start before 600 seconds, and the quiet period ends 120 seconds after the last update, at 670 seconds, so the run starts at 670 seconds.

Two requirements usually mean two settings: a cost cap needs the minimum time, "do not start in the middle of a burst" needs a short wait. Size the wait to the gap between changes within one delivery, not to the gap between deliveries: a wait longer than the usual gap between changes keeps postponing the run on a busy source.

A table update trigger starts a run when one or more monitored objects receive new data. It reads the objects' metadata and change history, so the job follows the data without being wired into the producer's job. Supported objects are Unity Catalog tables (managed and external Delta tables, and managed Iceberg tables), materialized views, streaming tables, views and metric views built on them. A view that uses read_files, or depends on tables outside Unity Catalog or on federated tables, is not supported: the trigger can be saved but never fires. This is the trigger for "start when another team's table has been refreshed", at times that team controls and does not publish, without changing their pipeline.

Any vs All

With several monitored tables, the Trigger when condition decides how they combine:

The cost of All: it waits for every monitored table, so any table in the list that no longer changes means the condition can never be met and the job stops running. Only tables that are expected to keep changing belong in an All condition.

Views count every change to their base tables

For a monitored view, any update to a table the view depends on counts as an update to the view, even when the view's result does not change. A view selecting country = 'DE' from a shipments table that receives frequent rows for other countries therefore triggers on every one of those writes. If runs must follow German shipments only, use a source that changes only with EU data or have the job check for new rows and exit early.

Run-as permissions

The trigger is evaluated as the job's Run as principal, which needs SELECT on every monitored object plus USE CATALOG and USE SCHEMA on its parents. For a monitored view it also needs SELECT on the view's underlying tables, even if it can query the view through the view owner's access. Without these privileges the trigger cannot evaluate and the job never runs, which shows up as an empty run history rather than failed runs. Write privileges, job permissions and BROWSE do not help.

The timing settings in the previous section apply to table update triggers too; the minimum time is counted from the completion of the previous run.

A continuous trigger keeps a job always running: exactly one run is active at a time, and a new run starts shortly after the previous one ends. Databricks recommends it for always-on Structured Streaming workloads, replacing the older pattern of a manually started job with unlimited retries and one concurrent run.

• Failure handling is at the job level. When runs fail repeatedly, Databricks waits longer between restarts (exponential backoff). You do not configure task retries for this.
• No task dependencies. A continuous job cannot have tasks linked with Depends on. For a bronze, silver and gold chain of streaming notebooks this is fine: each streams from the table above it, so the tasks can run in parallel; with dependencies the downstream tasks would wait forever, because an upstream stream never completes.
• Not combined with other triggers. A continuous job does not also have a schedule, file arrival or table update trigger.
• Serverless compute: only bounded Structured Streaming triggers such as trigger(availableNow=True) are supported in continuous jobs; time-based stream triggers (processingTime, continuous processing) are not. Each run processes what is available and ends, the continuous trigger starts the next run, and the checkpoint keeps processing exactly once.

Continuous mode suits work that must never stop. It is the wrong choice for a job that should run only when something changed, or once at a fixed time, because it reruns regardless.

Every trigger hands its runs to the same job-level controls, and they decide what happens when a run is due while another is still active.

• Maximum concurrent runs defaults to 1 for new jobs. A new run that would exceed the limit is skipped when queueing is off: it appears with a Skipped state (the Jobs API reports MAX_CONCURRENT_RUNS_EXCEEDED), and no task failed.
• Queueing, when turned on, holds such a run in a queue for up to 48 hours and starts it when capacity frees up. Runs are never merged. Queueing is on by default for jobs created in the UI after 15 April 2024, which is why an overrun on a newer job usually shows a queued run rather than a skipped one.

A queued run starts late, so anything that names output by "now" drifts; see the trigger time reference below.

Pause and resume

Pausing a trigger stops it from creating new runs while keeping its configuration; runs already in progress finish, and Resume restores the same schedule or condition. Manual runs keep working while a trigger is paused: Run now on a paused continuous job produces a single run, and the job stays paused once it finishes. Deleting the trigger instead loses its configuration, and lowering the concurrency limit to stop runs affects manual runs as well.

Dynamic value references pass facts about the run into task parameters, and the trigger references tell the code why and on what the run started.

The distinction between trigger time and start time matters with queueing: a daily run due at 23:50 that waits in the queue for 20 minutes still has the trigger time of its scheduled day, while its start time and any current_timestamp() in the code fall on the next day.

Choose a data-driven trigger when the job should follow the data's availability, and a scheduled trigger when the requirement is a point in time or there is nothing to observe. The questions to ask:

1. Is there a fixed cut-off or deadline? "One report at 06:00 from whatever has arrived" or "one report per weekday from the data as of 06:00" is time-based. Data-driven triggers would fire too early, several times, or never when one input is missing; late data simply goes into the next scheduled run.
2. Does the data arrive at unpredictable times, and should the job react promptly without wasting runs? Files in a volume or external location: file arrival. A table refreshed by another pipeline or team: table update. A schedule guessed from usual refresh times either runs before the data (and builds from stale data on late days) or adds delay on normal days.
3. Is there anything to observe? A job that fetches exchange rates from an external web service produces no files or upstream table changes for Databricks to watch, and a table update trigger on the table the job writes would wait for the job itself. When the source sends no signal, use a schedule.
4. Must it never stop? Always-on streaming: continuous.

Pipeline dependencies decide the configuration of a data-driven trigger: depend on the table you actually read (not on the producer's input files, which fire before the table is built), use All when every input must be fresh and Any when each change matters on its own, and add the timing settings for cost caps and bursts.