RAG (Retrieval-Augmented Generation) is a method where a language model first looks up matching passages in a knowledge base and receives them in the prompt together with your question. The model then answers based on those passages instead of relying only on its training knowledge.
Think of an open-book exam. Without RAG, the model has to answer everything from memory. With RAG, it may look in the right folder first. If you ask a shop's support bot "How long does the warranty last?", it first finds the matching paragraph in the warranty terms and builds its answer from that, including a note on where it found it.
Here you will learn how this works technically, what it costs compared with a huge prompt, where RAG fails, and when fine-tuning or a large context window is the better choice.
1. Why RAG exists
A language model only knows what was in its training data. Two things follow from that.
- It does not know your data. Internal manuals, contracts, customer tickets or your price list were never part of the training.
- Its knowledge ends at a cutoff date. It knows nothing about what happened afterwards. More on that in the entry on the knowledge cutoff date.
If you ask about it anyway, the model often fills the gap with something that sounds plausible but is wrong. That is called a hallucination. RAG targets exactly this problem. It gives the model the missing material before it answers. The term was coined in 2020 by a research team around Patrick Lewis from Meta AI, UCL and NYU.
2. How RAG works
A RAG system has two parts. The first runs once up front, the second with every question.
2.1 Up front: preparing the knowledge
The documents are split into sections, often a few hundred words long. For each section, an embedding model computes a vector of numbers that describes its meaning. Texts with similar content end up close together, even when they use different words. So "return" and "exchange" sit close together, while "return" and "billing address" sit far apart. The vectors go into a vector database or a search index.
2.2 With every question: search, enrich, answer
- Retrieval: The question is also turned into a vector. The search returns the sections most similar to it. Many systems combine this with a classic keyword search so that product numbers or proper names do not get lost.
- Augmentation: The best matches are written into the prompt together with the question, usually with the instruction to use only these sources.
- Generation: The language model writes the answer and, ideally, points to the sections it came from.
The key point: the model itself is not changed. RAG only changes what is in the prompt. That is why you can update the knowledge base at any time without retraining anything.
3. A worked example: a knowledge base of 3,000 pages
Take a mid-sized company with 3,000 pages of manuals, policies and FAQs. With the assumptions from the entry on the context window, that comes to about 1,800,000 tokens.
Of the 343 models in our directory, only 6 have a context window that can hold this amount at all. With RAG, you send only the 5 best sections per question, each about 300 words long. That is roughly 2,000 tokens, or about 0.1% of the knowledge base.
- Input tokens per question
- 1,800,000
- Input cost per question
- $3.60
- Input tokens per question
- 2,000
- Input cost per question
- $0.004
gradually.aiThe calculation uses the input price of Claude Sonnet 5.5, and the cost of the answer comes on top in both cases. For the price of a single question with the full knowledge base, RAG gets you about 900 answers. Prompt caching would make the first variant cheaper, but it does not solve the space problem.
This is what the prompt looks like that the model actually receives. The example is made up and does not come from a specific system:
Answer the question using only the following excerpts.
If the answer is not in them, say so.
Cite the source in square brackets at the end.
[Warranty terms, section 3.1]
We grant a 24-month warranty on all devices from the date of purchase.
Batteries are covered for 12 months.
[FAQ, returns]
You can return unused items within 30 days.
Question: How long does the warranty on my device's battery last?Example answer:
The battery is covered by a 12-month warranty from the date of purchase.
The rest of the device is covered for 24 months. [Warranty terms, section 3.1]The instruction "If the answer is not in them, say so" is no minor detail. Without it, the model happily falls back on its own knowledge when the search finds nothing, and mixes it in unnoticed.
4. RAG, fine-tuning or a large context?
The three routes solve different problems. This table helps you decide:
- Good for
- Large or frequently changing bodies of knowledge, answers with source citations, access rights per document
- Weak at
- Questions that need all the material at once, such as "Summarize all contracts"
- Good for
- A few documents that must be read in full, quick prototypes
- Weak at
- Large volumes, high cost per question, accuracy that drops in very long prompts
- Good for
- Style, tone, fixed formats, recurring tasks
- Weak at
- Storing new facts reliably and keeping them current
- Good for
- Current, public information
- Weak at
- Internal data and sources of varying quality
gradually.aiIn practice, many systems combine the routes. A fine-tuned model supplies the tone you want, RAG supplies the facts. When retraining is worth the effort is covered in the entry on fine-tuning.
5. Where RAG fails
RAG sounds like a simple cure for wrong answers. But the answer can only be as good as the search before it. A team around Scott Barnett analyzed three RAG projects and described seven typical failure points. The most important ones:
- The search does not find the right passage. The model then answers from the wrong matches or from none at all. This happens, for example, when the question uses different terms than the document.
- The passage is there, but cut apart. If the exception to a rule sits in the next section, the model sees only half the truth.
- The model does not use the matches properly. It overlooks the decisive line, mixes in its own knowledge or draws the wrong conclusions.
- The documents are outdated or contradict each other. If two versions of a policy sit in the index, the search may well find the old one.
A preregistered Stanford study from 2024 shows how large the effect can be. Vendors of legal research tools had advertised RAG and in some cases promised "hallucination-free" citations. In the test, the tools from LexisNexis and Thomson Reuters still hallucinated on 17 to 33% of queries. That was less than a general-purpose chatbot, but far from zero.
There is also a security risk. Whatever is in your knowledge base lands directly in the prompt. If a document or web page contains hidden instructions, the model may follow them. That is called prompt injection.
6. What this means for you
- Check first whether you need RAG. If your documents fit comfortably into the context window, just upload them. That is simpler and often more accurate. The token counter shows you how much fits.
- Use ready-made solutions before you build your own. Project folders in chatbots and the file search that OpenAI and Google offer in their APIs cover many cases without you running a vector database.
- Keep the knowledge base clean. Delete old versions, give documents clear titles and split them at sensible points, such as headings.
- Demand citations. Have the model back every statement with the section it came from, and spot-check the results.
- Test with real questions. Collect 20 to 50 typical questions with known answers and check whether the search finds the right passages. Only then is it worth refining the prompt.
- Allow "I don't know". Tell the model in the system prompt to admit missing information instead of guessing.