Standard RAG follows a fixed pipeline: take the user query, retrieve documents, generate an answer. This works well for straightforward questions, but it breaks down when the query is ambiguous, when a single retrieval pass is not enough, or when the retrieved documents are irrelevant.
Agentic RAG gives the LLM agency over the retrieval process itself. Instead of blindly retrieving and generating, an agentic RAG system lets the model decide whether retrieval is needed, rewrite the query for better results, evaluate whether the retrieved documents actually answer the question, and perform additional retrieval steps if they do not.
This turns the retrieve-then-generate pipeline into a reasoning loop: the agent observes the query, decides on an action (retrieve, rewrite, search a different source, or answer directly), evaluates the result, and iterates until it has enough evidence to generate a confident response. The patterns that enable this -- tool-use, self-reflection, and adaptive routing -- are what separate demo-quality RAG from production-quality RAG systems that handle the messy reality of user queries.
Raw user queries are often poor search queries. Conversational language, pronouns, and follow-up questions retrieve poorly because they lack the specific terms that match document content. Query rewriting is the highest-impact agentic RAG technique -- it improves retrieval quality without changing the index, the embedding model, or the chunking strategy.
Self-reflection adds a quality gate between retrieval and generation. Instead of blindly passing retrieved documents to the LLM, the agent evaluates whether they are relevant. If not, it rewrites the query and tries again. This catches cases where the initial query was too vague or used different terminology than the indexed documents. The max_retries cap prevents infinite loops.
Not every query should hit the vector store. Questions about structured data are better answered by SQL. Questions about current events need web search. Simple questions need no retrieval at all. Adaptive routing avoids wasting retrieval on queries that would get poor results from the vector store and directs each query to its best data source.
Complex questions often span multiple topics that are covered in different documents. A single retrieval may surface documents about one aspect but miss others. Decomposition retrieves evidence for each sub-question independently, then feeds all evidence to the LLM for synthesis. This dramatically improves answer completeness for comparison and multi-facet questions.
This is the most flexible agentic RAG pattern. The LLM has retrieval available as a tool and decides when to use it, what to search for, and how many times to search. It can call multiple tools in sequence, combine results from different sources, and decide on its own when it has enough information to answer. The while loop continues until the model stops requesting tool use.
Making the agent loop unbounded, allowing the model to perform unlimited retrieval steps. This leads to runaway costs, high latency, and occasionally infinite loops.
Cap the agent loop at 3-5 iterations. If the agent cannot find a satisfactory answer within that limit, return the best available response with a disclaimer. Always set max_retries or max_iterations.
Using a large, expensive model for routing and query rewriting decisions that do not require deep reasoning. Each agent decision adds latency and cost.
Use a small, fast model for routing (query classification) and query rewriting. Reserve the larger model for the final generation step where quality matters most. A small model can determine 'this is a SQL question' in under 100ms.
Not providing the agent with a 'no retrieval needed' path, forcing it to always retrieve even for simple factual questions the model can answer from its training data.
Include a direct_answer or skip_retrieval option in your routing logic. The agent should be able to answer 'What is Python?' without hitting the vector store. This saves latency and cost on easy queries.
Evaluating retrieval quality using only the similarity score from the vector database. High similarity does not guarantee relevance -- the retrieved chunk may be semantically similar but not actually answer the question.
Use LLM-based relevance evaluation (self-reflection) to judge whether retrieved documents genuinely answer the query. Similarity scores measure embedding-space distance, not answer quality.
Decomposing every query into sub-questions, even simple ones. This adds multiple LLM calls and retrieval steps to straightforward questions, increasing latency by 3-5x for no benefit.
Only decompose queries that are complex (comparisons, multi-facet questions, questions with multiple entities). Use a lightweight classifier to decide whether decomposition is necessary before invoking it.
Agentic RAG gives the LLM control over the retrieval process: when to search, what to search for, whether the results are good enough, and whether to try again. The key techniques are query rewriting (transforming user questions into better search queries), self-reflection (evaluating retrieval quality before generation), adaptive routing (choosing the right data source per query), and multi-step retrieval (decomposing complex questions). Each technique adds latency and cost, so add them incrementally based on evaluation results rather than building a complex agent from the start.
Interactive lessons and challenges on Stanza, practice in VS Code, Cursor, or the web.
Interactive lessons and challenges, right in your code editor.
Check the free courses. No credit card.