RAG — retrieval-augmented generation

A model only “knows” what it saw during training — nothing about your internal docs, your contracts, or what changed last week. RAG (Retrieval-Augmented Generation) fixes that without retraining anything: before answering, the system fetches the relevant passages from your data and hands them to the model along with the question.

The analogy

The open-book exam. Instead of demanding that the student memorize the whole syllabus (expensive, quickly outdated, and they’ll invent what they forgot), you let them bring the textbook. The skill shifts: it’s no longer “remember everything”, it’s find the right page fast and cite it.

The principle

flowchart LR
    Q([question]) --> E["embed the question"]
    E --> S["search the vector database"]
    S --> P["top relevant passages"]
    P --> LLM["LLM: question + passages"]
    Q --> LLM
    LLM --> A([sourced answer])

Your documents are cut into chunks and turned into embeddings — numbers that capture meaning — stored in a vector database.
At question time, the system finds the closest passages by meaning (not just keywords) and injects them into the prompt.
The model answers from the provided passages, ideally citing them — that’s grounding at work.

A concrete example

An HR assistant for a 400-page policy handbook:

Q: "How many days of parental leave do I get?"
retrieve → 3 passages from "Leave policy v12" (updated last month)
LLM      → "15 weeks, per the Leave Policy §4.2 [source],
            extended to 19 weeks for a second child [source]."

No fine-tuning, and updating the answer = updating the document — the next question already uses version 13.

When to use it

Answers must come from your data: internal docs, product catalogs, contracts, tickets.
Content changes often — re-indexing a document is instant, retraining a model is not.
You need citations: “here’s where that comes from” builds trust and catches errors.

When to avoid it

General-knowledge questions the model already handles — retrieval just adds cost and noise.
Questions that require reasoning over the whole corpus at once (“summarize all 2,000 tickets”) — that’s a parallelization job.

The classic trap

Blaming the model for a retrieval problem. If the search returns the wrong passages, the LLM confidently answers from the wrong page — it looks like a hallucination, but it’s a search bug. Log what was retrieved, and when RAG misbehaves, look at the passages before touching the prompt.