Streaming Data in Fabric: Eventstreams, Spark, and Real-Time Intelligence

Choose Eventstream when you need to capture, lightly transform, and route events without writing code; choose Spark structured streaming when you need complex, code-first transformations landing in lakehouse Delta tables; choose KQL in Real-Time Intelligence when the goal is interactive analytics, time-series exploration, dashboards, or alerting over high-volume event data. The three are complementary, not competing — a common pattern uses an Eventstream to ingest and route, an Eventhouse to store and query, and Spark only where custom logic demands it.

Exam scenarios usually hinge on two signals: the skill profile (no-code analysts point to Eventstream; data engineers writing Python point to Spark; teams fluent in KQL point to an Eventhouse) and the workload shape (routing points to Eventstream; heavy transformation points to Spark; interactive time-series analysis points to KQL).

Where custom Python fits

Python in a stream usually points to Spark, but be precise about why. KQL does have a python() plugin that runs inline Python in a sandbox inside the Eventhouse, which suits small per-query calculations. Libraries you publish to a Fabric environment (custom packages and their dependencies) load only into Spark sessions, so a requirement to run a packaged model or library against the stream is a Spark structured streaming job. Eventstream operators cannot run custom Python at all.

Non-streaming near-twins

Two Fabric items often appear as wrong answers in streaming scenarios. Dataflow Gen2 is refresh-based batch: it runs on a schedule and cannot subscribe to Azure Event Hubs or Azure IoT Hub as a continuous stream, so even a frequent refresh is not streaming ingestion. A Warehouse is not an Eventstream destination and is not a data source for Real-Time Dashboards; streaming data for dashboards belongs in an Eventhouse.

An Eventstream ingests events from a source, optionally transforms them in a no-code editor, and routes the results to one or more destinations. It is the front door for streaming data in Fabric.

Sources

Eventstreams connect to Azure Event Hubs, Azure IoT Hub, Kafka-compatible endpoints, change data capture feeds from operational databases, Fabric workspace and job events, sample data for testing, and a custom endpoint your own applications can publish to.

Transformations

The event processor supplies drag-and-drop operators:

• Filter drops events that fail a condition.
• Manage fields renames, casts, adds, or removes columns.
• Aggregate and Group by compute metrics over time; Group by is where you configure tumbling, hopping, sliding, session, and snapshot windows.
• Expand flattens an array field: it produces one output row per array element, repeating the event's other fields on each row.
• Union combines streams that share a schema into one stream; it does not match rows on a key.
• Join matches events on a key — either two streams within a time window, or a stream against reference data (see below).
• A SQL operator lets you express the transformation as a SQL-style query instead of chaining visual operators.

Enriching a stream with reference data

The Join operator can enrich each event from a Lakehouse Delta table added as a reference data source — for example, adding each device's site and owner from a device registry to its readings. The reference source can be refreshed periodically on an interval you set, so changes to the table reach the enrichment without restarting the stream; with no refresh, the stream keeps using the data it loaded. Pick the join type by what should happen to unmatched events: an inner join drops events with no matching reference row, while a left outer join keeps them with empty enrichment columns. A Union is not a substitute — it appends rows, it does not look anything up.

Destinations

One Eventstream can route the same source to several destinations at once (fan-out), so you never re-ingest a source to feed a second consumer. There is no Warehouse, Dataflow Gen2, or Power BI semantic model destination.

Derived streams and the Real-Time hub

A derived stream is the named output of a transformation chain. It is published to the Real-Time hub, where other teams can discover it and subscribe to it or route it onward, so one raw feed can fan out into several shaped feeds. Connecting an operator straight to a destination does the same transformation but publishes no stream — nobody else can find or reuse that shaped feed.

Eventstream transformation is deliberately lightweight. When a scenario needs custom code, packaged Python, stateful deduplication, or joins against large historical tables, route the stream to a lakehouse or Eventhouse and do the heavy work in Spark or KQL.

Use a native table when you ingest events into the Eventhouse itself and need the fastest query performance; use a OneLake shortcut when the data already lives elsewhere in OneLake (for example, Delta tables a Spark job maintains in a lakehouse) and you want to query it from KQL without copying it.

A native table stores data in the Eventhouse's own optimized columnar format with hot caching and indexing. Everything KQL is built for — sub-second aggregations over billions of rows, time-series operators, update policies, materialized views — performs best against native tables, and data ingested from an Eventstream is queryable within seconds. If you also enable OneLake availability on the table, the Eventhouse writes a Delta copy into OneLake so Spark, SQL, and Power BI can read the same data without export jobs; that setting exposes native data outward, which is the reverse direction from a shortcut.

A OneLake shortcut points the KQL database at external Delta or Parquet data in a lakehouse, warehouse, or another OneLake location. Nothing is duplicated and there is no ingestion pipeline to maintain — the shortcut always reflects the current state of the source. The trade-off is performance: standard shortcut queries read the external files at query time, without the Eventhouse's native indexing and caching, so they run noticeably slower than native-table queries.

The decision rule: data born in the stream and queried constantly belongs in native tables; data owned and maintained by another engine that KQL only needs to reference — dimension tables, historical archives, lakehouse gold tables — belongs behind a shortcut.

Enable Query acceleration on a OneLake shortcut when you need shortcut convenience — no data duplication, no ingestion pipeline — but query performance close to a native table; keep a standard shortcut when queries are occasional and the cost of acceleration is not justified.

With a standard shortcut, every query reads the underlying Delta or Parquet files directly at query time. That is fine for infrequent lookups or joins against small reference data, but it is slow for the interactive, high-frequency query patterns Real-Time Dashboards depend on.

Query acceleration changes the physics: the Eventhouse indexes and caches the shortcut data in the background, so queries against the accelerated shortcut behave much like queries against native tables — while the source remains the single copy of the data, still owned and updated by the producing engine.

The hot caching period is the cost lever

The query acceleration policy sets a hot period: how many days of the source data are indexed and cached. Acceleration consumes capacity in proportion to the data it keeps hot, so size the period to the range your queries actually read — if dashboards read the last 30 days of a five-year table, a 30-day hot period gives them the speed without paying to cache years nobody queries. Accelerate the shortcuts that are queried frequently; leave rarely used lookups standard.

Limits to know

• One copy, one owner. Neither shortcut option copies the data, so there is no second copy to reconcile with the source; ingesting into a native table does create one.
• Accelerated shortcuts behave like external tables. You cannot build a materialized view over one. If you need precomputed aggregates, ingest the data into a native table (for example, by adding an Eventhouse destination to the Eventstream) and build the view there.
• Update policies: an update policy whose source is an accelerated delta shortcut is supported only in preview, and it runs asynchronously and non-transactionally, without processing deletes made in the Delta table. Even then it transforms each batch independently, so it cannot maintain running aggregates — that is a materialized view's job.
• Freshness is not seconds. New data reaches an accelerated shortcut only after the source write lands (a Lakehouse destination batches its Delta commits) and the background indexing catches up. Acceleration is for fast interactive queries over data owned elsewhere, not for alerting that must fire within seconds — alerting data belongs in native Eventhouse tables.

Spark structured streaming treats a live data source as an unbounded table: you define a query with readStream and writeStream, and Spark runs it incrementally as micro-batches, applying your transformations to new data as it arrives. In Fabric you run it from a notebook or Spark job definition, typically reading from Azure Event Hubs, a Kafka endpoint, or a Delta table an Eventstream is feeding, and writing Delta tables in a lakehouse.

from pyspark.sql.functions import window, col

df = (spark.readStream
      .format("delta")
      .load("Tables/raw_events"))

agg = (df.withWatermark("eventTime", "10 minutes")
         .groupBy(window(col("eventTime"), "5 minutes"), "deviceId")
         .count())

(agg.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "Files/checkpoints/device_counts")
    .toTable("device_counts"))

Checkpointing

The checkpointLocation is where Spark persists the query's progress: the source offsets already processed, the commit log of completed micro-batches, and the state store (running window aggregates, deduplication keys). It is not the Delta transaction log, which belongs to the target table. After a failure or restart, Spark resumes from the checkpoint, and together with the Delta sink this gives exactly-once output.

The checkpoint must be a stable path, unique to that query, reused on every run. If the path changes per run (built from a timestamp, say) or the folder is deleted, Spark sees no history and reprocesses the source from the beginning — an append query then writes duplicate copies of rows already in the table. A streaming write without a checkpoint location is the classic wrong answer.

Watermarks and late data

withWatermark("eventTime", "10 minutes") tells Spark how late, in event time, data may arrive. Data arriving within the delay threshold is guaranteed to be aggregated into its window; data later than the threshold may be dropped (it is not guaranteed to be). Once the watermark passes a window's end, Spark can finalize that window and evict its state, so the state store stops growing. Set the threshold to the lateness you actually see — too short drops valid events, too long delays results and holds more state. A longer trigger interval or a processing-time window does not handle late events; event-time windows plus a watermark do.

Output modes and the Delta sink

Three consequences follow:

• A query with no aggregation — filter, select, write — uses append mode, the simplest configuration.
• In append mode, the watermark delay sets result latency: a window appears in the table only after the watermark passes its end, so a 1-hour watermark on 15-minute windows holds each result back by more than an hour. Append mode on an aggregation without a watermark is rejected.
• The Delta streaming sink supports only append and complete. To keep a large table current with changing aggregates (per-key totals that keep changing, for example) without rewriting it each trigger, use foreachBatch and run a Delta MERGE that upserts each micro-batch.

Stateful operations

dropDuplicates on a stream keeps the keys it has seen as state in the checkpoint; pair it with a watermark so that state is bounded. cache() is not supported on streaming DataFrames. Choose structured streaming when the scenario says custom transformation logic, Python or Scala packages, joins with large Delta tables, stateful deduplication, or medallion-style refinement of a stream — things Eventstream operators cannot express.

KQL processes streaming data after it lands in an Eventhouse: you aggregate, reshape, and analyze events with query operators, and you can transform data at ingestion time with update policies or maintain pre-computed aggregates with materialized views. You write and save queries in a KQL queryset; a Real-Time Dashboard is a separate item whose tiles each hold their own KQL query against a data source such as an Eventhouse database (you can pin a queryset query to a dashboard as a tile). Activator alerts can also be driven from KQL queries.

Time bucketing: bin() vs make-series

The workhorse pattern is summarize with bin(), which rounds each timestamp down to the start of its interval for grouping — effectively a tumbling window over stored data:

Telemetry
| where Timestamp > ago(1h)
| summarize avgTemp = avg(Temperature), events = count()
    by DeviceId, bin(Timestamp, 5m)

summarize ... by bin() returns only bins that contain data: an hour with no errors simply has no row, and sorting or aligning bins with bin_at() does not bring it back. make-series builds a regular series per group and fills empty bins with a default value (such as 0). Give it an explicit from ... to range; without one, the series spans only the first to last bin that has data, so empty leading or trailing bins are still missing.

Telemetry
| make-series avgTemp = avg(Temperature) default = 0
    on Timestamp from ago(1d) to now() step 15m by DeviceId
| extend anomalies = series_decompose_anomalies(avgTemp)

The series_* functions operate on the arrays make-series produces, not on summarize rows. series_decompose_anomalies() decomposes each series into seasonality, trend, and residual before flagging outliers, so a regular daily peak is not reported as an anomaly — unlike a fixed threshold such as "three times the average".

Update policies

An update policy runs a KQL function automatically whenever a batch is ingested into a source table, writing the transformed rows to a target table — useful for parsing raw JSON into typed columns as it arrives. Key rules:

• The policy is defined on the target table and names its source. To split one raw table into two typed tables, put one policy on each target, both reading the same source.
• IsTransactional defaults to false: if the policy fails, ingestion into the source still succeeds and the target misses that batch. Set it to true so the source and target ingestion succeed or fail together.
• To transform and discard raw rows, set the source table's retention soft-delete period to 0 seconds; this requires a transactional policy, so data is never lost between the tables.
• Each batch is transformed independently — an update policy cannot maintain aggregates across batches.

Materialized views

A materialized view keeps an aggregation continuously up to date so dashboards read a pre-aggregated result instead of scanning raw events. Its query must aggregate (a summarize). The classic latest-record-per-key view uses arg_max(): summarize arg_max(Timestamp, *) by DeviceId. Querying the view by name combines the materialized part with records not yet materialized, so results are fully current. materialized_view("ViewName") reads only the materialized part — cheaper, but possibly slightly stale; materialized_view("ViewName", 10m) reads only the materialized part when it is at most 10 minutes old and otherwise falls back to the full view. The view's lookback property (for arg_max, arg_min, and take_any views) limits how far back deduplication looks; it does not control how stale query results may be.

Near-twins that do not precompute

Reach for KQL when the scenario emphasizes interactive exploration of high-volume telemetry, time-series operators, near-real-time dashboards, or log analytics.

Windowing functions divide an unbounded stream into finite time buckets so you can aggregate it; the four you must distinguish are tumbling, hopping, sliding, and session windows. Scenarios describe a timing requirement and ask for the window.

Boundaries and emission

A tumbling window is simply a hopping window whose hop equals its size. For a hopping window, the size is the lookback and the hop is how often a result is produced: "the last 60 minutes, every 15 minutes" is size 60, hop 15. Tumbling and hopping windows are aligned to the clock, so a burst can straddle a boundary: five errors between 09:15:24 and 09:15:38 split two and three across the 30-second tumbling windows starting 09:15:00 and 09:15:30, so neither window reaches a threshold of four. A sliding window evaluates whenever an event enters or leaves, so it catches the condition in any 30-second span. A session window has no fixed size at all — it opens with an event and closes after a configured inactivity timeout, and it should be grouped by the entity (user, device) whose activity defines the session.

Implementing windows in each engine

• Eventstream: the Group by operator offers tumbling, hopping, sliding, and session windows, plus snapshot windows (events sharing the same timestamp).
• Spark: window(col, windowDuration, slideDuration, startTime). With only a duration it is tumbling; a slide shorter than the duration creates overlapping windows (Spark calls them sliding windows; they behave like hopping windows), so each event is counted in duration ÷ slide windows — window("eventTime", "10 minutes", "5 minutes") counts every event twice. startTime only offsets the window boundaries. session_window(col, gapDuration) builds inactivity-gap sessions; group by the user key too (groupBy(session_window("ts", "20 minutes"), "userId")) or one session spans all users' events. Pair both with withWatermark.
• KQL: bin() gives tumbling behaviour over stored data. row_window_session(Timestamp, MaxDistanceFromFirst, MaxDistanceBetweenNeighbors[, Restart]) runs over serialized (time-ordered) rows and returns, for each row, the start time of its session; a new session starts when the gap to the previous row exceeds the limit, the session reaches its maximum length, or the restart condition is true (for example, when the user changes). The session_count plugin only counts active sessions over time; it does not assign rows to sessions.

Here is how the pieces compose for a typical requirement: thousands of factory sensors publish temperature readings; operations needs a live dashboard and an alert whenever a device's five-minute average exceeds a threshold, and data science wants the raw history in the lakehouse.

1. Ingest: the sensors publish to Azure Event Hubs (or Azure IoT Hub). An Eventstream uses it as a source — no code, just a connection.
2. Shape and route: a Manage fields operator casts the payload into typed columns. The Eventstream routes the full raw stream to a Lakehouse destination as Delta tables for data science, and the shaped stream to an Eventhouse destination set to event processing before ingestion — one source fanning out to two destinations.
3. Analyze: in the Eventhouse, the readings land in a native table (hot, constantly queried streaming data — not a shortcut). A Real-Time Dashboard tile runs summarize avg(Temperature) by DeviceId, bin(Timestamp, 5m) against it — tumbling-window behaviour.
4. Alert: Activator watches the stream or the KQL query and fires a Teams message or email when the five-minute average crosses the threshold. Because the data is native, the alert sees readings within seconds.
5. Reference data: the device master table is maintained in the lakehouse by another team, so the KQL database reaches it through a OneLake shortcut; if dashboard joins against it prove slow, enable Query acceleration rather than building an ingestion copy.

Notice what was not needed: Spark. It enters only if the requirement adds custom logic — say, scoring readings with a Python package from a Fabric environment — at which point a structured streaming job with readStream from the lakehouse Delta table, a stable checkpoint location, and append output mode does that one job.