What is big data? The three Vs explained
Big data refers to datasets so large, fast-moving or varied that traditional tools and a single machine cannot store or process them practically — a category usually characterised by the "three Vs": volume, velocity and variety. The term describes a scale problem rather than a specific technology: once data outgrows what one database server or one analyst’s spreadsheet can handle, the tools and techniques have to change. Cloud computing is what made working at that scale affordable for most organisations, by renting elastic storage and processing power instead of buying and running a data centre. This guide explains the three Vs plainly, why the cloud changed what was practical, the core approaches to storing and processing data at scale, and where the topic shows up in data-focused certification study.
The three Vs, explained plainly
Volume is the most obvious one: big data means a genuinely large quantity of data, more than fits comfortably on one machine or in one traditional database. Velocity describes how fast that data arrives — sometimes in a steady stream, sometimes in real time, such as sensor readings or clickstream events, rather than arriving in a single batch once a day. Variety describes the shape of the data: big data is rarely one tidy table. It mixes structured data (rows and columns), semi-structured data (JSON or log files with some organisation but no fixed schema), and unstructured data (text, images, video) that has no inherent structure at all.
Some definitions add a fourth or fifth V — veracity, meaning how trustworthy and clean the data is, and value, meaning whether the effort of processing it actually produces something useful. Both are worth keeping in mind: a large, fast, varied dataset that is unreliable or never analysed is not actually an asset, it is just a cost.
Why the cloud made big data practical
Before elastic cloud infrastructure, working at big-data scale meant buying enough servers to handle your largest expected workload and running them permanently, whether you needed that capacity or not. The cloud changed the economics: you can rent storage and processing power for the scale of a specific job, use it for as long as the job runs, and release it afterwards, paying only for what you used. That shift is what turned big-data analysis from something only the largest technology companies could afford into a standard part of many organisations’ toolkits.
The core approaches: distributed and batch vs streaming
Two ideas underpin almost every big-data system. The first is distribution: instead of one machine doing all the storage and computation, the data and the work are split across many machines that operate in parallel, so the system scales by adding more machines rather than by buying a bigger one. The second is the choice between batch and streaming processing — batch collects data and processes it in scheduled chunks, which suits reporting and analytics that do not need to be instantaneous, while streaming processes each event as it arrives, which suits use cases like fraud detection or live dashboards. Our explainer on ETL pipelines covers how data is extracted, transformed and loaded in more detail, including where batch and streaming fit into that pipeline.
Where big data is stored and processed
Big data typically lands in a data lake — a large, flexible store that accepts data in its raw, original format — and is later organised into a data warehouse for structured analysis, or queried directly from the lake. Our explainer on data lakes vs data warehouses covers the difference between those two models in full. Distributed processing frameworks such as Hadoop and Spark are the classic open-source tools for running computation across many machines at once, and every major cloud provider offers managed services that implement the same ideas without you having to run the clusters yourself.
- AWS — EMR for managed distributed processing, and Redshift as a cloud data warehouse.
- Azure — Synapse Analytics and the newer Fabric platform, combining data integration, warehousing and analytics.
- Google Cloud — BigQuery as a serverless data warehouse, and Dataproc for managed Hadoop and Spark clusters.
How big data connects to analytics, ML and data engineering
Big data is rarely the end goal — it is the raw material for analytics dashboards, business reporting and machine learning models, all of which need large, well-organised datasets to produce reliable results. The discipline of building and maintaining the pipelines that collect, clean and move that data is data engineering, and it has become a distinct career path in its own right as the volume and variety of data organisations collect has grown.
Where big data appears in certification study
Data-focused certifications test big-data concepts directly — the AWS Data Engineer Associate exam covers distributed processing, storage choices and pipeline design at length, and Microsoft’s Fabric Data Engineer Associate exam covers the equivalent ground on Azure’s unified analytics platform. Even broader exams such as AWS Solutions Architect Associate expect you to recognise when a workload calls for a data lake or a managed analytics service rather than a conventional database. Understanding the three Vs and the batch/streaming distinction is the conceptual foundation those exams build on.
Original practice questions, timed mock exams and revision notes. No card, nothing to pay.