Why does AI need GPUs? Parallel maths explained

CPU versus GPU: general-purpose versus parallel

A CPU is a general-purpose processor. It has a small number of powerful cores that are excellent at handling varied, step-by-step tasks quickly: running your operating system, a database, a web server. It is built to do many different kinds of work well.

A GPU, or graphics processing unit, has a very large number of simpler cores designed to do the same operation on many pieces of data at the same time. It was originally built to draw graphics, which means calculating millions of pixels in parallel. That same strength turned out to be exactly what AI needed.

Why neural-network maths suits parallel hardware

A neural network is, at heart, a huge collection of numbers arranged in grids, called matrices, that are multiplied and added together over and over. Each of those multiplications is independent of many others, so they can be done simultaneously rather than waiting in a queue.

That is why a GPU fits so well. Spreading one enormous batch of identical, independent calculations across thousands of cores finishes far sooner than working through them in order. Our explainer on large language models shows how many such operations sit inside a single response.

Training versus inference

The two phases of a model’s life place different demands on hardware:

• Training — the model repeatedly processes vast amounts of data and adjusts itself. It is by far the most GPU-hungry phase, often needing many GPUs working together for a long time.
• Inference — using the finished model to answer a request. It needs far less computing per request, but runs constantly for every user, so it adds up at scale and benefits from fast hardware too.

Other specialised chips

GPUs are not the only option. Providers also build custom accelerators designed specifically for machine-learning workloads, such as Google’s TPUs (tensor processing units), and AWS and others offer their own AI-focused chips. The principle is the same: hardware built around parallel matrix maths beats general-purpose hardware for this job.

Which chip is best depends on the model, the budget and the provider, and the details move quickly, so check the provider’s documentation rather than relying on a snapshot. The concept that specialised parallel hardware accelerates AI is the durable part.

What this means in the cloud

Few organisations buy GPUs outright. Most rent accelerated computing from a cloud provider and pay for the time they use it, which is much of the appeal of the cloud for AI. Managed services such as Amazon SageMaker handle much of the infrastructure so teams can focus on the model, and our explainer on what Amazon SageMaker is covers that.

To see how the cost of training and running models differs, read our explainer on training versus inference. For exam preparation, our /revision library covers the infrastructure side of AI lesson by lesson.