Large language models know a lot, but they have a hard cutoff: they only know what was in their training data. Ask about your company's internal docs, yesterday's product release, or a niche regulatory filing, and the model either hallucinates or admits ignorance. Retrieval-Augmented Generation (RAG) solves this by injecting relevant context into the prompt at query time.
The pattern is straightforward: when a user asks a question, you first search a knowledge base for documents that are likely to contain the answer, then pass those documents alongside the question to the LLM. The model generates a response grounded in the retrieved evidence rather than relying solely on memorized training data. This means you can update knowledge without retraining, cite sources for every answer, and keep proprietary data out of the model's weights.
RAG has become the default architecture for question-answering over private data, customer support bots, internal search tools, and any application where factual accuracy and up-to-date information matter more than creative generation.
This builds a complete retrieval pipeline without any paid APIs. sentence-transformers runs locally, FAISS provides in-memory vector search, and the L2 normalization converts inner product to cosine similarity. In production, you would replace FAISS with a persistent vector database like pgvector or Qdrant.
The system prompt constrains the model to only use the provided context, reducing hallucination. Document numbering in the context enables the model to cite specific sources. This two-step pattern -- retrieve then generate -- is the foundation of every RAG system.
LangChain orchestrates the full pipeline: loading, chunking, embedding, storing in Chroma, and chaining retrieval with generation. Chroma is open-source and runs as an embedded database with no server required. The persist_directory ensures embeddings survive process restarts.
pgvector turns PostgreSQL into a vector database. The HNSW index makes similarity search fast (sub-millisecond for millions of vectors). The <=> operator computes cosine distance. This approach keeps your vectors in the same database as your application data, eliminating the need for a separate vector database service.
This is not runnable code but a structured decision framework. The key insight is that RAG and fine-tuning solve different problems: RAG provides knowledge, fine-tuning changes behavior. Most teams should start with RAG because it ships faster, costs less, and handles the most common use case -- grounding LLM responses in your own data.
Stuffing entire documents into the prompt without chunking, which exceeds the context window or dilutes the relevant information with irrelevant content.
Split documents into focused chunks of 200-1000 tokens. Smaller, targeted chunks lead to better retrieval precision because the embedding represents a coherent unit of meaning rather than a sprawling document.
Using the same embedding model for queries and documents when the model was not trained for asymmetric search, leading to poor retrieval quality.
Use embedding models explicitly trained for retrieval (e.g., sentence-transformers/all-MiniLM-L6-v2, nomic-embed-text). These models are trained on query-document pairs and handle the asymmetry between short queries and longer passages.
Returning too many or too few retrieved documents. Too many floods the context with noise; too few risks missing the relevant passage.
Start with k=3 to 5 retrieved chunks and tune based on your use case. Implement a relevance score threshold to filter out low-similarity results rather than always returning a fixed number.
Not telling the LLM to only use the provided context, which lets the model blend retrieved facts with hallucinated information.
Include explicit instructions in the system prompt: 'Answer using ONLY the provided context. If the context does not contain enough information to answer, say you do not know.' This constrains the model to the evidence.
Treating RAG as a one-time setup and never evaluating retrieval quality, leading to degraded performance as the knowledge base grows and changes.
Build an evaluation dataset of question-answer pairs with known source documents. Measure retrieval recall (did the correct document appear in top-k?) and answer faithfulness (does the answer match the source?) on every pipeline change.
Retrieval-Augmented Generation grounds LLM responses in external knowledge by retrieving relevant documents before generating an answer. The pattern is simple: embed the query, search a vector store for similar chunks, inject them into the prompt, and let the model generate a response constrained to the evidence. RAG ships faster than fine-tuning, handles changing knowledge gracefully, and provides source citations for every answer. The quality of the retrieval step determines the quality of the entire system -- invest in chunking strategy, embedding model selection, and relevance filtering before optimizing the generation side.
Interactive lessons and challenges on Stanza, practice in VS Code, Cursor, or the web.
Interactive lessons and challenges, right in your code editor.
Check the free courses. No credit card.