What is a data pipeline? Moving data from source to use
A data pipeline is an automated series of steps that moves data from its sources to a destination — ingesting, processing and delivering it — so that raw data becomes usable for analytics, reporting or machine learning. Rather than someone manually exporting a spreadsheet and cleaning it by hand, a pipeline runs the same journey reliably and repeatedly, on a schedule or continuously, so the data landing in a dashboard or a model is always current and consistently shaped. This guide covers the plain idea, how it relates to ETL, batch versus streaming pipelines, the components involved, and where pipelines appear in data engineering certification study.
The plain idea
Data rarely arrives ready to use. It starts scattered across application databases, log files, third-party services and sensors, in whatever format each source happens to produce. A data pipeline is the automated path that data takes from those sources to somewhere useful: it is ingested (collected from source systems), transformed (cleaned, reshaped, combined, validated) and stored or served in a form that analysts, dashboards or machine learning systems can consume directly.
The value is in the automation and repeatability. A pipeline runs the same defined steps every time, on the same schedule or the same trigger, so the data feeding a report or a model has a known, consistent lineage rather than depending on someone remembering to redo a manual process correctly.
How this relates to ETL
ETL — extract, transform, load — is one well-known pattern of data pipeline, covered in full in our what is ETL explainer, and it is worth not repeating that detail here. The relationship is that ETL is a specific sequence (extract, then transform, then load) within the broader idea of a data pipeline; pipelines also cover patterns ETL does not name directly, including ELT (load first, transform afterwards, inside the destination) and continuous streaming flows. Every ETL job is a data pipeline; not every data pipeline follows the ETL order.
Batch versus streaming, conceptually
Pipelines generally move data in one of two rhythms. A batch pipeline collects data and processes it in bulk at intervals — hourly, nightly, or on some other schedule — which suits use cases where a delay of hours is acceptable, such as a daily sales report. A streaming pipeline processes data continuously, as each event arrives, which suits use cases where freshness matters within seconds, such as fraud detection or live operational dashboards. Many real systems combine both, batching some data and streaming other data through the same overall architecture.
The components of a pipeline
A data pipeline is typically assembled from a few recurring parts:
- Sources — the systems data originates from: application databases, event logs, third-party APIs, sensors.
- Ingestion — the layer that collects data from those sources, whether in batches or as a continuous stream.
- Transformation — cleaning, validating, reshaping and combining data into a consistent, usable form.
- Storage — a data lake for raw and semi-structured data, or a data warehouse for structured, query-ready data (see our data lake versus data warehouse explainer for how the two differ).
- Orchestration — the layer that schedules each step, manages dependencies between them, and monitors whether a run succeeded, so failures are caught rather than silently producing stale or incomplete data.
Why pipelines matter, and data quality
Reliable, repeatable data movement is the backbone that analytics and machine learning sit on: a dashboard is only as trustworthy as the pipeline feeding it, and a machine learning model is only as good as the data it was trained and served on — see our explainers on MLOps and big data for how pipelines connect into those wider practices. Because a pipeline runs unattended, checks for data quality — validating schemas, catching missing or malformed records, alerting when a run behaves unexpectedly — need to be built into the pipeline itself rather than caught by a person reviewing the output afterwards.
Cloud services and certification study
All major clouds offer managed pipeline building blocks rather than requiring you to run the infrastructure yourself: AWS offers services including Glue for data integration and transformation, Step Functions for orchestrating multi-step workflows, and Kinesis for streaming ingestion; Azure offers Data Factory and Fabric for building and orchestrating pipelines; Google Cloud offers Dataflow for batch and streaming data processing. Data pipelines are a central topic in the AWS Certified Data Engineer Associate (DEA-C01) exam, covering ingestion, transformation and orchestration choices, and in the Microsoft Certified: Fabric Data Engineer Associate (DP-700) exam, which focuses on building pipelines within the Fabric platform. The Developer Associate exam expects a lighter, conceptual awareness of how application code might produce or consume pipeline data via events and queues.
Original practice questions, timed mock exams and revision notes. No card, nothing to pay.