Streaming Data in Fabric: Eventstreams, Spark, and Real-Time Intelligence

Choose Eventstream when you need to capture, lightly transform, and route events without writing code; choose Spark structured streaming when you need complex, code-first transformations landing in lakehouse Delta tables; choose KQL in Real-Time Intelligence when the goal is interactive analytics, time-series exploration, dashboards, or alerting over high-volume event data. The three are complementary, not competing — a common pattern uses an Eventstream to ingest and route, an Eventhouse to store and query, and Spark only where custom logic demands it.

Exam scenarios usually hinge on two signals: the skill profile (no-code analysts point to Eventstream; data engineers writing Python point to Spark; log-analytics queries point to KQL) and the workload shape (routing points to Eventstream; heavy transformation points to Spark; interactive time-series analysis points to KQL).

An Eventstream ingests events from a source, optionally transforms them in a no-code editor, and routes the results to one or more destinations. It is the front door for streaming data in Fabric.

Sources

Eventstreams connect to Azure Event Hubs, Azure IoT Hub, Kafka-compatible endpoints, change data capture feeds from operational databases, Fabric workspace and job events, sample data for testing, and a custom endpoint your own applications can publish to.

Transformations

The event processor supplies drag-and-drop operators: Filter to drop events, Manage fields to rename, cast, or remove columns, Aggregate and Group by to compute rolling metrics — Group by is where you configure tumbling, hopping, sliding, session, and snapshot windows — plus Union, Expand (flatten arrays), and Join (combine two streams on a key within a time window).

Destinations and derived streams

Route the output to an Eventhouse (KQL database) for real-time analytics, a Lakehouse as Delta tables for downstream batch work, a custom endpoint for external consumers, or Activator to fire alerts directly from the stream. A derived stream is the named output of a transformation chain: it materializes the transformed events as a new stream that other consumers can subscribe to or that you can route onward, so one raw feed can fan out into several shaped feeds without re-ingesting the source.

For DP-700, remember that Eventstream transformation is deliberately lightweight. When a scenario needs custom code, machine learning, or joins against large historical tables, route the raw stream to a lakehouse or Eventhouse and do the heavy work in Spark or KQL.

Use a native table when you ingest events into the Eventhouse itself and need the fastest query performance; use a OneLake shortcut when the data already lives elsewhere in OneLake (for example, Delta tables a Spark job maintains in a lakehouse) and you want to query it from KQL without copying it.

A native table stores data in the Eventhouse's own optimized columnar format with hot caching and indexing. Everything KQL is built for — sub-second aggregations over billions of rows, time-series operators, update policies, materialized views — performs best against native tables. If you also enable OneLake availability on the table, the Eventhouse writes a Delta copy into OneLake so Spark, SQL, and Power BI can read the same data without export jobs; that setting exposes native data outward, which is the reverse direction from a shortcut.

A OneLake shortcut points the KQL database at external Delta or Parquet data in a lakehouse, warehouse, or another OneLake location. Nothing is duplicated and there is no ingestion pipeline to maintain — the shortcut always reflects the current state of the source. The trade-off is performance: standard shortcut queries read the external files at query time, without the Eventhouse's native indexing and caching, so they run noticeably slower than native-table queries.

The decision rule the exam probes: data born in the stream and queried constantly belongs in native tables; data owned and maintained by another engine that KQL only needs to reference — dimension tables, historical archives, lakehouse gold tables — belongs behind a shortcut.

Enable Query acceleration on a OneLake shortcut when you need shortcut convenience — no data duplication, no ingestion pipeline — but query performance close to a native table; keep a standard shortcut when queries are occasional and the cost of acceleration is not justified.

With a standard shortcut, every query reads the underlying Delta or Parquet files directly at query time. That is fine for infrequent lookups or joins against small reference data, but it is slow for the interactive, high-frequency query patterns Real-Time Dashboards and alerting depend on.

Query acceleration changes the physics: the Eventhouse indexes and caches the shortcut data in the background, so queries against the accelerated shortcut behave much like queries against native tables — while the source remains the single copy of the data, still owned and updated by the producing engine. New data appearing in the source Delta table becomes queryable shortly after it lands, without you building any ingestion.

The trade-offs to weigh:

• Cost: acceleration consumes additional capacity for the background indexing and caching work; standard shortcuts do not.
• Freshness: accelerated data appears after a short indexing delay rather than at the exact moment of the source write.
• Ownership: both options leave the source system in charge of the data; neither creates a second copy you must reconcile, which is the key difference from ingesting into a native table.

Exam trigger phrasing: query lakehouse data from KQL without ingesting or copying it, with near-native performance points at Query acceleration; minimize cost for rarely queried external data points at a standard shortcut; maximum performance for hot streaming data points at a native table.

Spark structured streaming treats a live data source as an unbounded table: you define a query with readStream and writeStream, and Spark runs it incrementally as micro-batches, applying your transformations to new data as it arrives. In Fabric you run it from a notebook or Spark job definition, typically reading from Azure Event Hubs, a Kafka endpoint, or a Delta table an Eventstream is feeding, and writing Delta tables in a lakehouse.

from pyspark.sql.functions import window, col

df = (spark.readStream
      .format("delta")
      .load("Tables/raw_events"))

agg = (df.withWatermark("eventTime", "10 minutes")
         .groupBy(window(col("eventTime"), "5 minutes"), "deviceId")
         .count())

(agg.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "Files/checkpoints/device_counts")
    .toTable("device_counts"))

Checkpointing

The checkpointLocation stores stream progress and state. It is what makes the stream fault-tolerant: after a failure or restart, Spark resumes from the checkpoint instead of reprocessing or skipping data. A streaming write without a checkpoint location is the classic exam wrong answer.

Output modes

Append writes only new finalized rows (the default, required for plain inserts and for windowed aggregations with watermarks once a window closes). Complete rewrites the entire aggregation result every batch — only valid for aggregations. Update writes only rows that changed since the last batch. The withWatermark call bounds how long Spark waits for late events, letting it finalize windows and drop old state instead of growing it forever.

Choose structured streaming when the scenario says custom transformation logic, Python or Scala, joins with historical Delta data, or medallion-style refinement of a stream — things Eventstream operators cannot express.

KQL processes streaming data after it lands in an Eventhouse: you aggregate, reshape, and analyze events with query operators, and you can transform data at ingestion time with update policies or maintain pre-computed aggregates with materialized views. Queries live in a KQL queryset and power Real-Time Dashboards and Activator alerts.

The workhorse pattern is summarize with bin(), which buckets events into fixed time intervals — effectively a tumbling window over stored data:

Telemetry
| where Timestamp > ago(1h)
| summarize avgTemp = avg(Temperature), events = count()
    by DeviceId, bin(Timestamp, 5m)

For time-series analysis, make-series builds a regular series per group, which the series_* functions then operate on — including built-in anomaly detection:

Telemetry
| make-series avgTemp = avg(Temperature) default = 0
    on Timestamp from ago(1d) to now() step 15m by DeviceId
| extend anomalies = series_decompose_anomalies(avgTemp)

Two server-side transformation tools matter for the exam. An update policy runs a KQL function automatically whenever new data is ingested into a source table, writing the transformed rows to a target table — the Eventhouse equivalent of an in-flight transformation, useful for parsing raw JSON into typed columns as it arrives. A materialized view keeps an aggregation (for example, latest-record-per-device or hourly averages) continuously up to date so dashboards query the pre-aggregated result instead of scanning raw events.

Reach for KQL when the scenario emphasizes interactive exploration of high-volume telemetry, time-series operators, near-real-time dashboards, or log analytics — the phrase analyze millions of events with sub-second queries is a Real-Time Intelligence signal, not a Spark one.

Windowing functions divide an unbounded stream into finite time buckets so you can aggregate it; the four you must distinguish are tumbling, hopping, sliding, and session windows. DP-700 tests them mostly as scenario matching — read the requirement, name the window.

A tumbling window is simply a hopping window whose hop equals its size. A sliding window differs from hopping in when it emits: hopping emits on a fixed schedule even for identical results, sliding emits only at moments the window contents change. A session window has no fixed size at all — it opens with an event and closes after a configured inactivity timeout.

Where you configure them: the Eventstream Group by operator offers all four (plus snapshot windows, which group events sharing the same timestamp); Spark structured streaming uses window() for tumbling and sliding-interval aggregations and session_window() for sessions, paired with withWatermark; in KQL, bin() gives tumbling behavior over stored data and row_window_session() assigns session identifiers based on gaps between records.

Here is how the pieces compose for a typical exam-style requirement: thousands of factory sensors publish temperature readings; operations needs a live dashboard and an alert whenever a device's five-minute average exceeds a threshold, and data science wants the raw history in the lakehouse.

1. Ingest: the sensors publish to Azure Event Hubs (or Azure IoT Hub). An Eventstream uses it as a source — no code, just a connection.
2. Shape and route: a Manage fields operator casts the payload into typed columns. The Eventstream routes the full raw stream to a Lakehouse destination as Delta tables for data science, and the same shaped stream to an Eventhouse destination for analytics — one source fanning out to two destinations.
3. Analyze: in the Eventhouse, the readings land in a native table (hot, constantly queried streaming data — not a shortcut). A KQL query using summarize avg(Temperature) by DeviceId, bin(Timestamp, 5m) — tumbling-window behavior — feeds a Real-Time Dashboard tile.
4. Alert: Activator watches the stream or the dashboard query and fires a Teams message or email when the five-minute average crosses the threshold.
5. Reference data: the device master table is maintained in the lakehouse by another team, so the KQL database reaches it through a OneLake shortcut; if dashboard joins against it prove slow, enable Query acceleration rather than building an ingestion copy.

Notice what was not needed: Spark. It enters only if the requirement adds custom logic — say, scoring readings against an ML model — at which point a structured streaming job with readStream from the lakehouse Delta table, a checkpoint location, and append output mode does that one job.