What is retrieval-augmented generation (RAG)?

The problem RAG solves

A language model knows only what was in its training data. That data was gathered at some point in the past — the knowledge cutoff — and it almost certainly never included your organisation’s policies, product documentation, tickets or contracts. Ask a plain model about any of that and the best case is “I don’t know”; the worse and common case is a hallucination — a fluent, confident answer assembled from plausible patterns rather than facts, which our explainer on large language models covers in depth.

For a general-purpose chat assistant this is tolerable. For an assistant answering staff questions about company policy, or customers’ questions about your product, it is disqualifying: the answers must come from the actual documents, be current, and ideally show their sources. That gap — a fluent generalist that needs to answer from specific, trusted, changing material — is exactly what RAG closes.

How RAG works, step by step

The idea is an open-book exam instead of a closed-book one. Rather than making the model answer from memory, the system fetches the relevant pages first and lets the model write its answer from them:

1. A user asks a question — say, “what is our parental-leave policy for contractors?”.
2. The system searches a prepared knowledge source — your documents, wiki, manuals — for the passages most relevant to the question.
3. The retrieval commonly uses embeddings and vector search: text is converted into numerical representations of meaning, so the search finds passages that are about the same thing even when the wording differs.
4. The best-matching passages are inserted into the model’s prompt as context, alongside the original question and an instruction to answer from the provided material.
5. The model generates its answer grounded in those passages — often citing which documents it drew on, so the reader can verify.

Why RAG became so popular

RAG’s appeal is that it fixes the biggest practical problems without touching the model itself. There is no retraining: the model stays exactly as the provider shipped it, and all the organisation-specific knowledge lives in an ordinary, inspectable document store. That makes the knowledge instantly updatable — correct a policy document and the very next question is answered from the corrected version, something no amount of model training can match for freshness.

It also makes answers more trustworthy in a verifiable way. Because the model is instructed to answer from retrieved passages, hallucination is substantially reduced — and because the system knows which passages it retrieved, it can cite them, turning “trust me” into “here is the source”. Add that sensitive documents remain in your own store under your own access controls rather than being baked into a model, and it is easy to see why RAG has become the default first architecture for building assistants over private knowledge — and why cloud services, including Amazon Bedrock with its managed knowledge-base features, offer it off the shelf.

The honest limits

RAG is only ever as good as two things: the documents in the store and the retrieval step that searches them. If the knowledge source is wrong, outdated or missing the answer, the model will faithfully generate a grounded — but wrong or empty — response; retrieval adds truthfulness only when there is truth to retrieve. And if the search surfaces the wrong passages, the model answers from irrelevant material. Questions whose answers are scattered across many documents, or that require reasoning over the whole collection rather than a few passages, remain genuinely hard for the retrieve-then-read pattern.

There are engineering realities too. Documents must be prepared and split into sensible chunks, kept in sync as they change, and access-controlled so the assistant cannot retrieve material the asking user should not see. Retrieval adds a step to every question, and grounding reduces hallucination without abolishing it — a model can still misread or over-summarise the passages it was given. None of this undermines RAG; it just means the retrieval side deserves as much care as the generation side.

RAG versus fine-tuning

The other established way to specialise a model is fine-tuning: taking a trained model and training it further on your own examples so that its internal parameters shift towards your domain. The two techniques are often presented as rivals, but they solve different problems. Fine-tuning is about behaviour — teaching the model a tone, a format, a specialised vocabulary or a task shape it should exhibit every time. RAG is about knowledge — giving the model current, specific facts at the moment of answering.

That difference drives the practical trade-offs. Fine-tuned knowledge is frozen at training time and updating it means training again, while RAG knowledge updates the moment a document does; fine-tuning requires curated training data and evaluation effort, while RAG requires search infrastructure and document hygiene; fine-tuning cannot cite sources, while citation is one of RAG’s core strengths. A fair summary: for answering from changing, checkable facts, start with RAG; for consistent style or task behaviour, consider fine-tuning; and mature systems sometimes use both — a fine-tuned model for behaviour, retrieval for facts.

RAG in certification study

RAG sits squarely in the generative-AI portion of the AWS Certified AI Practitioner (AIF-C01). The exam expects you to explain what RAG is and which problems it addresses — hallucination, knowledge cutoff, private data — to understand the roles of embeddings, vector search and knowledge bases at a conceptual level, and to weigh RAG against fine-tuning for a described scenario, which is precisely the comparison above. Amazon Bedrock’s knowledge-base capability is the AWS-flavoured expression of the pattern, worth recognising by name.

For developers, the AWS Certified Developer – Associate is natural next context: implementing RAG in practice is application engineering — wiring retrieval into prompts, handling sources and permissions, composing managed services — rather than data science. Either way, the concept layer in this article is the durable part: services and model line-ups change continually, but retrieve-then-generate, and the reasons it works, are stable ground.