Skip to content

What Is RAG? Retrieval-Augmented Generation Explained

Glowing document pages flowing from a bookshelf into a bright orb of light on a deep blue background


Glowing document pages flowing from a bookshelf into a bright orb of light on a deep blue background

RAG, short for retrieval-augmented generation, is a way of letting a language model look up facts in an outside collection of documents before it writes an answer.

What is RAG in practice? A search step in front of a language model. A model on its own can only use what it learned during training, and your private files were never part of it. The system finds matching passages, hands them to the model, and the model answers from that material. The idea comes from a 2020 paper by Patrick Lewis and colleagues, who paired a language model with a searchable index of Wikipedia.

What is RAG, in plain words

People often describe RAG as an open-book exam for AI. A plain model answers from memory, while a RAG system opens the book first. The book can be a wiki or a help center.

The flow has three moves. The system retrieves relevant passages, builds a prompt from the question plus those passages, and the model writes an answer grounded in the retrieved text. Indexing runs earlier: documents are split into short passages, turned into searchable numbers, and stored for fast lookup.

The generator can be any capable language model, such as GPT-4o. The retrieval step does the new work.


A stack of blank papers, a magnifying glass over glowing dots, and a speech bubble connected by a flowing line

Embeddings and vector databases do the finding

An embedding is a list of numbers that stands for the meaning of a piece of text. Passages about the same topic end up with number lists that sit close together, even when they share few exact words. A refunds question can match a returns policy page, though the words differ.

A vector database stores those number lists alongside the original text. It is built to answer one question fast: which stored passages sit nearest to this new one? The usual yardstick is cosine similarity, a measure of the angle between number lists. Close angle, close meaning.

When a question arrives, the system embeds it with the same model that embedded the documents, finds the nearest passages, and passes them to the generator. AWS describes the same shape: create embeddings to ingest documents into a vector database, then search that store each time a question arrives. Gemini 3 is one more model that can play the generator role.


Clusters of glowing dots connected by thin lines floating in dark blue space

Chunking decides what the model can find

Documents are too long to hand over whole. A forty-page manual would swamp the prompt, and most of it would be irrelevant to one question. So documents are cut into chunks, short passages retrieved one at a time.

Where the cuts fall matters more than most people expect. Cut too small and a chunk loses the sentence that explains it. Cut too large and the useful sentence drowns in unrelated text. Overlapping neighbors keeps a split rule whole in one chunk.

There is no chunk size that fits every kind of document. Tables and scanned pages split badly, and nested clauses have their own ways of breaking. The same problem shows up in tools that answer from your own notes, including Notion AI, which must find the right passage in a workspace full of pages before it can write anything useful.


A long paper scroll divided into equal colored segments on a pale background

Where RAG gets used in real products

Enterprise search was the first big home for RAG. An employee asks about parental leave, and the system answers from the current policy document instead of from a model that never saw it.

Support chatbots work the same way over help centers and manuals. A customer asks why a device keeps restarting, and the bot answers from the troubleshooting pages.

Web-grounded chat uses the same retrieve-then-write idea over search results, as the ChatGPT Search guide explains. Agents such as ChatGPT Agent lean on retrieval too. The Claude Sonnet guide covers one more model for the generator role.


A support agent wearing a headset working at a laptop with blank document pages floating around the desk

Why RAG cuts down on made-up answers

A model that must answer from memory will sometimes invent a confident reply. Give it the right passage and the job changes: it summarizes text it can see, which leaves less room for invention.

Freshness is the other win. A model trained months ago knows nothing about a policy changed last week. With RAG, you update the document store and the next answer reflects the change. No retraining run. AWS notes the same trade: updating stored documents costs far less than retraining a model from scratch.

Google Cloud describes this as grounded generation, where combining your data with a language model makes output more accurate, up to date, and relevant to the task at hand. The model still writes. It writes from evidence.


Rows of glowing translucent cubes on shelves with one bright cube lifted out

What RAG cannot fix

A retriever ranks passages by similarity, and similarity is not the same as relevance. A chunk can sit near the question in meaning and still be the wrong passage. If the right chunk never makes the shortlist, the model never sees it. The answer fails quietly. Retrieval quality sets the ceiling for everything after it.

The model can still drift even when retrieval succeeds. It may add details from general knowledge or bridge gaps between chunks with invented sentences. Research has shown models pay less attention to passages buried mid-prompt, so stuffing in more context can backfire.

Cost and care grow with the system. Every question pays for an embedding call and a search, and longer prompts cost more tokens. Access control must hold, or one employee could retrieve documents meant for someone else. IBM describes RAG as an architecture that connects a model to external knowledge bases so it can deliver more relevant responses, and that connection is only as good as the knowledge base behind it. The best AI tools compared rounds up several options built on these ideas.


A magnifying glass over scattered jigsaw puzzle pieces with one piece missing

Frequently asked questions about RAG

What does RAG stand for?

RAG stands for retrieval-augmented generation: retrieve passages first, then generate an answer from them.

How is RAG different from fine-tuning?

RAG adds context at answer time and leaves the model untouched. Fine-tuning updates the model’s weights with extra training. RAG suits knowledge that changes often. Fine-tuning suits fixed behavior and style.

Does RAG stop hallucinations completely?

No. RAG lowers the rate by giving the model real passages, but retrieval can miss and the model can still add details not in the sources. Treat citations as a check, and verify claims that matter.

What is a vector database in RAG?

A vector database stores embeddings, the number lists that stand for each passage’s meaning, and finds the ones nearest to a question.

What are embeddings in simple terms?

An embedding turns a piece of text into a list of numbers that captures what it means. Similar meanings get similar numbers, so search follows meaning instead of exact words.

What is chunking in RAG?

Chunking is the splitting of long documents into short passages before indexing. Good chunks hold one idea with enough context to stand alone.

Do I need RAG if my chatbot already searches the web?

Web search covers public pages, but it cannot reach your private documents. RAG searches a store you control, company policies or personal notes, and grounds answers in that material.

Leave a Comment