Retrieval Augmented Generation (RAG)

Large language models know a lot, but they don't know your company's internal data. They also make things up when they're unsure. RAG solves both problems by adding a retrieval step before generation.

Here's how it works. A user asks a question. The system converts that question into a vector embedding and searches a vector database for the most relevant documents. Those documents get passed to the LLM as context, along with the original question. The LLM then generates an answer based on the retrieved content instead of relying only on its training data.

The result is answers grounded in your actual documents, policies, product specs, or whatever you have indexed. This matters for enterprise use cases where accuracy is non-negotiable. A support agent pulling from your knowledge base needs to cite real policies, not generate plausible-sounding ones.

RAG architectures vary in complexity. A basic setup has an embedding model, a vector database, and an LLM. More advanced systems add re-ranking (scoring retrieved documents for relevance with a cross-encoder), chunking strategies (splitting documents into the right-sized pieces), hybrid search (combining BM25 keyword scoring with semantic similarity), and query rewriting (expanding or reformulating the user's question before retrieval).

Compared to fine-tuning, RAG is faster to set up and easier to update. When your data changes, you re-index the documents. No model retraining needed. The tradeoff: RAG only helps when the answer exists in your corpus. If the model's reasoning style or output format is wrong, fine-tuning is the right fix. If your corpus is small or your users ask broad questions, retrieval quality becomes the bottleneck and no amount of prompt tuning will save you.

RAG also isn't always the cheapest option. Each query runs an embedding call, a vector search, optional re-ranking, and a large context prompt. At high volume, that stacks up. Teams that measure cost per query often cache embeddings, cap top-k, and drop re-rankers on low-stakes queries.

A pharmaceutical sales rep asks a compliance assistant, "What's the recommended dosing for 65-year-old patients on our new antihypertensive?" The rep's question is embedded into a 3072-dim vector using OpenAI's text-embedding-3-large.

The system queries a Pinecone index holding 12,000 chunks from FDA-approved drug labels, clinical study reports, and internal medical affairs memos. Top-k=20 chunks come back by cosine similarity. A Cohere Rerank call scores those 20 against the original question and keeps the top 5, which cover the geriatric dosing table, renal adjustment guidance, and a cardiology sub-study.

Those 5 chunks, plus the question and a system prompt that says "answer only from the provided sources and cite each claim," go to Claude Sonnet. The model returns a 120-word answer citing section 2.3 of the approved label for the 5mg starting dose and section 8.5 for the renal caution. A post-processor confirms each cited passage actually appears in the retrieved chunks. The rep gets a grounded answer in about 1.8 seconds. No hallucinated dosage, and a compliance audit trail is logged automatically.