What Is Retrieval-Augmented Generation (RAG)? A Plain-English Guide

A large language model knows a great deal about the world in general and nothing about your company's refund policy, last month's board minutes or the manual for the machine on your factory floor. Retrieval-augmented generation, usually shortened to RAG, is the most common way to close that gap. Instead of retraining the model, you give it the relevant documents at the moment a question is asked.

The idea in one paragraph

When someone asks a question, a RAG system first retrieves the passages from your documents that are most likely to contain the answer. It then passes those passages to the language model along with the question and an instruction such as "answer using only this material". The model generates an answer grounded in what was retrieved, ideally with references back to the source.

How it works, step by step

1. Collect the documents. Policies, help-centre articles, contracts, product specs, meeting notes.
2. Split them into chunks. Long documents are divided into sections of a few hundred words so the system can pull out just the relevant part.
3. Create embeddings. Each chunk is converted into a list of numbers, called an embedding, that represents its meaning. Passages about similar ideas end up with similar numbers even if they use different words.
4. Store them in a searchable index, often called a vector database.
5. At question time, the question is converted into an embedding too, and the index returns the closest-matching chunks. Many systems combine this with traditional keyword search for better results.
6. Generate the answer. The model reads the question plus the retrieved chunks and writes a response, ideally citing which chunk each point came from.

Why not just retrain the model?

• Freshness: update a document and the next answer reflects it. Retraining is slow and expensive.
• Traceability: answers can link to the exact source passage, so people can check them.
• Access control: retrieval can respect permissions, so an employee only gets answers from documents they are allowed to see.
• Cost: indexing documents is far cheaper than fine-tuning a model for every change.

Good use cases

• Internal help desks that answer "how do I..." questions from policy and process documents
• Customer support assistants grounded in a help centre
• Searching large collections of contracts, research or technical manuals
• Drafting answers to repetitive questionnaires using previous approved responses

Where RAG still goes wrong

RAG reduces made-up answers but does not eliminate them. The common failure points are predictable:

• Bad retrieval. If the right passage is not retrieved, the model answers from the wrong one or from general knowledge. Most quality problems start here, not in the model.
• Messy source documents. Outdated, duplicated or contradictory documents produce outdated or contradictory answers. RAG makes your document hygiene visible.
• Poor chunking. Splitting a table or a numbered procedure in the middle loses meaning.
• Over-trusting fluent answers. A well-written response with a citation can still misread the source. See our guide to checking AI output for hallucinations.

Questions to ask before buying a RAG product

1. Does every answer show its sources, and can users open them?
2. How are document permissions enforced?
3. How quickly do changes to source documents appear in answers?
4. What happens when no relevant document exists? A good system says so.
5. Where is our data stored and processed, and is it used to train anyone's models?
6. How will we measure answer quality over time?

Our checklist for evaluating an AI tool before adoption covers the privacy and testing questions in more depth.

Start small

The most successful RAG projects start with one well-maintained document set and a clear group of users, such as an HR policy library for staff questions. Build a list of 30 to 50 real questions with known correct answers, test against it, and expand only when the answers are reliably right. Clean documents and honest evaluation matter more than the choice of model.