What Is RAG (Retrieval-Augmented Generation), and When Should Devs Actually Use It?

Image: LangChain documentation, RAG concepts page, used for editorial coverage. The pipeline diagram below is by Neural Tech Daily.

The short answer

RAG (Retrieval-Augmented Generation) is the architectural pattern where a large language model answers questions by first retrieving relevant chunks from a vector database, then generating an answer grounded in those chunks. The pattern was introduced by Lewis et al. in a 2020 Facebook AI Research paper ¹ and has since become the default production pattern for LLM apps that need to reason over private or up-to-date data.

For developers, RAG is the right pattern when (a) the corpus updates frequently, (b) the corpus is too large to fit in even a 1M-token context window, or (c) cost matters and re-embedding 10,000 documents beats paying per-token for full-context calls. Skip RAG when the corpus is under fifty documents and stable (just put it in the system prompt) or when the answer requires reasoning across the full corpus rather than retrieval of specific facts. Fine-tuning and RAG solve different problems, and teams confuse them often.

What RAG actually is

RAG is a three-step pipeline. The pipeline runs in two phases: an index-time phase that happens once per document update, and a query-time phase that runs on every user query.

At index time, documents are split into chunks (typically 200 to 1,000 tokens each), each chunk is converted to a vector embedding by an embedding model, and the vectors are stored in a vector database with metadata pointing back to the source. At query time, the user’s query is embedded by the same embedding model, the vector database returns the top-k most similar chunks by cosine similarity, and the chunks are concatenated into the LLM’s prompt as context. The LLM then generates an answer grounded in those chunks. ²

The “retrieval” piece is borrowed from classic information retrieval. The “augmented generation” piece is what’s new: instead of the LLM relying on whatever facts it memorised during training, it gets the relevant facts injected into its prompt at query time. This means the LLM can answer questions about documents it has never seen, as long as those documents are in the vector database.

The key insight is that the LLM never reads the full corpus. It reads only the chunks the retriever surfaced. That’s a feature when the corpus is too big to fit in context, and a bug when the answer requires synthesising across many chunks the retriever couldn’t find.

How a typical RAG pipeline works

Diagram by Neural Tech Daily.

The diagram above shows what most production RAG stacks look like. The popular building blocks for each layer are well-documented: LangChain ² and LlamaIndex are the orchestration frameworks; Chroma ³ , Pinecone ⁴ , Qdrant, and Weaviate are the vector databases; OpenAI’s text-embedding-3, Cohere’s embed-english-v3.0 (and the newer embed-v4 flagship), and open-weight models like BGE and E5 are the embedding models. The exact stack matters less than the discipline of getting each step right, which is mostly about chunking strategy and retrieval evaluation.

When to use RAG

Three use cases justify the RAG complexity, and a useful filter is whether any of these three describes the actual problem.

The first is when the corpus updates frequently. A documentation site that ships new pages weekly, an internal wiki that engineers edit daily, or a product catalogue that changes with stock all need an architecture where adding a document means re-embedding one document, not re-training a model. Fine-tuning is the wrong tool here; the cost of retraining each week is too high.

The second is when the corpus is too large to fit in even a 1M-token context window. A 10,000-document corporate knowledge base will not fit in any current model’s context, even Gemini’s 2M-token configurations ⁷ or Claude’s 1M-token configuration ⁶ . Retrieval is the only way to surface the relevant chunks. Long-context models help with the synthesis step (more chunks fit in the prompt), but retrieval still does the filtering work.

The third is when cost matters and the alternative is paying per-token for repeated full-context calls. If an FAQ bot serves 10,000 queries a day and each query needs the full 200-page product manual, the LLM API bill on a long-context approach is large. Embedding the manual once, retrieving the relevant 2,000 tokens per query, and paying for the smaller prompt is typically much cheaper at volume; the exact ratio depends on the model’s input-token pricing and the retrieval-vs-full-context token ratio for the specific corpus.

For a dev team building an internal AI tool, the most common right-fit case is the first or second: customer-support knowledge bases, codebases, regulatory documents, or product specs that are too large or too dynamic for a system-prompt approach.

When NOT to use RAG

Two cases where teams reach for RAG and shouldn’t.

The first is a small, stable corpus. Under 50 documents, where each document is under 5,000 tokens, the simpler approach is to put the whole corpus in the system prompt or retrieve the entire corpus on every query. No vector database, no chunking strategy, no retrieval evaluation. Many internal-tool projects fit this shape and would ship sooner without the RAG infrastructure overhead.

The second is when the answer requires reasoning across the full corpus, not retrieval of specific facts. “Summarise the company’s overall hiring strategy across all 200 internal memos” is not a retrieval problem; it is a synthesis problem. A long-context model that ingests all 200 memos in one call will outperform a RAG system that retrieves the top-10 chunks and tries to synthesise from there. RAG’s failure mode here is that the retriever surfaces semantically similar chunks but misses chunks that are relevant for structural reasons (e.g., chronological context).

There is a third case worth flagging: when fine-tuning is what the team actually wants. Fine-tuning teaches the model new behaviour or style; RAG gives the model new facts. A team that wants the model to write in the company’s voice should fine-tune, not RAG. A team that wants the model to know last week’s product updates should RAG, not fine-tune. Teams that conflate the two end up with neither working well.

RAG vs fine-tuning: different problems

Question	RAG fits	Fine-tuning fits
Need to add new factual knowledge?	Yes	Possible, expensive, brittle
Need to teach new behaviour or style?	No	Yes
Corpus changes weekly or faster?	Yes	No, retraining is too costly
Need answers grounded in specific documents?	Yes (chunks are citable)	No (model memorises facts opaquely)
Need lower per-query latency?	RAG adds retrieval latency	Fine-tuning adds nothing at query time
Need to update on a schedule?	Re-embed changed docs	Retrain the model

The mental model is: RAG is for facts, fine-tuning is for skills. A model fine-tuned to write SQL for the company’s schema will still hallucinate the latest schema changes; a RAG pipeline that retrieves the schema docs will get the latest schema right but won’t write SQL any better than the base model. Most production systems need both for different reasons.

RAG vs long-context: a real trade-off

Image: Anthropic engineering blog, Contextual Retrieval, used for editorial coverage of the indexing-stage preprocessing technique discussed below.

By 2026, every major frontier-tier model from OpenAI, Anthropic, and Google offers long-context configurations. Older Gemini configurations reach 2M; the current Gemini 2.5 Pro flagship offers 1M with 2M planned ⁷ . Claude offers 1M-token mode ⁶ . GPT-class models routinely handle 128K to 200K. The question for builders is whether long-context replaces RAG.

The answer depends on three factors: corpus size, query volume, and budget. A useful rule-of-thumb is that a small-to-mid corpus with low query volume often fits comfortably in long-context calls, and the simpler architecture wins. A corpus that exceeds the model’s context limit, or query volume that pushes the per-token bill into uncomfortable territory, pushes back to retrieval economics, where embedding once and retrieving cheaply beats paying per-token for full-context calls on every query.

The interesting middle case is what Anthropic calls “Contextual Retrieval” ⁵ , which operates at the indexing stage rather than at prompt time: each chunk is augmented with LLM-generated explanatory context (50 to 100 tokens) before embedding and BM25 indexing, with an optional reranking pass at retrieval time. Anthropic reports a 49% reduction in retrieval failures from the contextual-embeddings plus contextual-BM25 combination alone, and 67% with reranking added on top. This is preprocessing, not “retrieve more chunks for a richer prompt”; the retrieval-time chunk count in Anthropic’s experiments stays in the standard 5-to-20 range. For most production stacks in 2026, Contextual Retrieval is the realistic answer when retrieval recall is the bottleneck.

Common RAG failure modes

Four ways teams ship RAG and watch it underperform.

Bad chunking. Splitting a 50-page document into 500-token chunks at arbitrary boundaries breaks semantic units. A definition that spans a paragraph break gets split across two chunks; the retriever surfaces one half of the definition and the LLM hallucinates the other half. The fix is semantic chunking — splitting at section boundaries, preserving overlapping context windows, and storing chunk-level metadata about position and parent document.

Embedding mismatch. The embedding model used at index time and the one used at query time must be the same model. Teams sometimes upgrade the embedding model and forget to re-embed the corpus, leaving the vector database in an inconsistent state where new query embeddings don’t match old document embeddings. The retriever appears to work but surfaces irrelevant chunks.

Missing metadata. Chunks without metadata (source URL, document title, section heading, last-updated date) make grounding harder. The LLM sees text without context and can’t cite the source. Teams that skip metadata at index time end up bolting it on later, which is more expensive than doing it once at the start.

Hallucination despite retrieval. Even with relevant chunks in the context, LLMs sometimes answer from training-data memory rather than from the retrieved chunks. The fix is prompt discipline (“answer only from the provided context, otherwise say ‘I don’t know’”) plus evaluation. Production RAG systems need an evaluation harness that measures retrieval recall, answer faithfulness, and answer relevance separately. Without that, the team cannot diagnose whether failures come from bad retrieval or bad generation.

How to learn RAG hands-on

Reading explainers gets a developer to “I understand what RAG is.” Building one gets them to “I understand which design choices matter.” The second is what production work needs.

For self-paced study, the LangChain RAG concepts page ² and Pinecone’s RAG overview ⁴ are the two best starting points. Anthropic’s Contextual Retrieval write-up ⁵ is the right next read once the basics are clear, especially for teams that need to ship at scale. The original Lewis et al. paper ¹ is worth reading after the practical foundations are in place; it grounds the modern stack in the research origin.

Honest caveats

Three caveats worth flagging for any team adopting RAG in 2026.

The vector-database vendor space is crowded and the differentiation is thinner than vendors suggest. Chroma is the easy local-development choice; Pinecone is the managed-service default; Qdrant and Weaviate offer self-hostable alternatives. For most projects, the embedding model and chunking strategy matter more than the vector database choice. Don’t let vendor selection block the project.

Long-context models are closing the gap on small-to-mid corpora faster than the RAG ecosystem expected. A team starting a new project in 2026 should benchmark a long-context approach against a RAG approach before defaulting to either; for corpora that fit in a model’s available context, the simpler architecture often wins on time-to-ship and operational complexity.

RAG is not a substitute for evaluation infrastructure. Teams that ship RAG without an evaluation harness ship a system they cannot improve. Building the evaluation harness alongside the pipeline is the single highest-payoff discipline for production RAG work.