RAG in AI means retrieval augmented generation: a system first finds relevant material in an external source, then gives that material to a language model to help it answer. For example, a college assistant could retrieve a current examination notice before explaining an application rule. The source lookup is the retrieval; combining it with the question supplies the extra context for generation.
Microsoft's RAG documentation describes the pattern as grounding a language model's responses in an organisation's own content. IBM's explanation makes the distinction from retraining explicit: external information can be supplied when answering, rather than first being built into a new round of model training. Those are two different ways of adapting AI to a task.
| Stage | What happens |
|---|---|
| Question | The user asks for information |
| Retrieval | The system searches an allowed knowledge source |
| Context | Relevant passages accompany the question |
| Generation | The language model produces an answer |
| Checking | The reader can inspect any supplied sources |
How does RAG work? A simple college notice example
Consider this fictional example, which does not describe a real college or deadline. A student asks whether a late examination application is accepted. The institution has a general prospectus, an examination notice and a later correction. A useful retrieval system needs to find the material that answers this particular question, including the correction if it changes the rule.
The application can place the relevant passages alongside the student's question and ask a language model to explain them. The resulting answer might identify the applicable condition and point to the notice. It should not invent a deadline if the retrieved material does not state one. That final safeguard is a design requirement in this example, not a guarantee built into the letters RAG.
A conventional search engine generally returns material for a person to read. In a RAG application, a language model also uses the retrieved material to compose a response. A fluent explanation can be helpful, but the underlying passage still matters. A student should be able to open the notice and check whether the answer has preserved its conditions, exceptions and scope.
Why are documents split into chunks?
A handbook or policy archive can be much larger than the material needed to answer one question. Microsoft describes chunking as dividing large documents into portions that can be matched independently. Instead of sending an entire archive, the application can select smaller passages. That makes the retrieval decision central: the model can only use the evidence actually provided to it at that stage.
Search can use keywords, vector representations or a combination. Vectors are numerical representations used to find material by similarity; they are not a separate source of verified facts. Microsoft's documentation describes hybrid search as combining keyword and vector queries. Ranking then helps determine which results reach the answering stage. A familiar word in a document is not enough if the passage concerns a different rule or department.
Is RAG the same as a context window or fine tuning?
A context window is the amount of material a model can process in a request, while RAG is a way of selecting information to place into that request. Microsoft's documentation highlights limited token inputs as a retrieval challenge. A larger allowance does not automatically identify which notice is current or which paragraph answers the question. Selection and capacity solve different problems.
Fine tuning involves further training a model on a particular dataset. RAG instead lets the application look up information from an external source. IBM explains that the approaches can also be used together. For a reader choosing an application, the useful question is whether the answer is drawing on the right documents, not whether a product description uses one term more often than another.
RAG also differs from an AI agent. Retrieval describes the information lookup and response pattern; an agent can involve tools and a wider workflow. Microsoft's current documentation includes both a classic retrieval pattern and an agentic approach that can break a question into multiple searches. Our AI agent explainer covers that separate concept.
Can a RAG answer still be wrong?
Yes. IBM states that RAG can reduce hallucination risk but cannot make a model error proof. A system might retrieve an outdated document, omit a relevant exception or generate an interpretation the source does not support. Citations are useful because they give readers a route back to the evidence; they do not replace reading it.
Access controls matter too. Microsoft identifies security and governance as core RAG challenges: an application should retrieve only content the user is allowed to see. For a public information task, ask which sources are included, how they are refreshed and whether the cited passage actually supports the answer. Our NotebookLM study guide and Claude Projects guide offer related reading about working with source material in AI applications.
