Chunking is the process of breaking long documents into smaller pieces before embedding and indexing them for retrieval. It is arguably the most impactful decision in a RAG pipeline, yet it receives far less attention than model selection or prompt engineering.
Why does chunking matter? Embedding models compress a piece of text into a single fixed-length vector. If you embed an entire 50-page document as one vector, that vector must represent everything in those 50 pages -- it becomes a vague average of all topics covered. When a user asks a specific question, the vague document-level vector may not be close enough to the query vector to surface in search results. But if you chunk that document into 100 focused paragraphs, the paragraph that actually answers the question will have a precise embedding that matches the query well.
The challenge is finding the right granularity. Chunks that are too small lose context. Chunks that are too large dilute relevance. Overlap strategies, separator hierarchies, and semantic boundaries all play a role in getting this right.
Fixed-size chunking is the most predictable strategy. The overlap parameter ensures that information at chunk boundaries appears in both the preceding and following chunk, so retrieval does not miss content that spans a split point. This is your baseline -- try this first before more complex strategies.
RecursiveCharacterTextSplitter is the most widely used chunking strategy in production RAG systems. It tries to split on paragraph boundaries first (double newline), then sentences (period-space), then words, and only falls back to character-level splitting if needed. This preserves the natural structure of the text while respecting the chunk size limit.
Character-based chunking can exceed embedding model token limits because tokens and characters do not have a fixed ratio. tiktoken (OpenAI's tokenizer) lets you split on actual token boundaries. This is critical when using models with strict token limits -- input beyond the limit is silently truncated, silently degrading embedding quality.
Semantic chunking creates boundaries where the topic shifts, producing chunks that are coherent units of meaning. It embeds each sentence and measures similarity between consecutive sentences -- when similarity drops below the threshold, a new chunk begins. This produces better retrieval results for documents that cover multiple topics but is slower due to the per-sentence embedding step.
There is no universal best chunk size. This evaluation framework measures retrieval recall at different chunk sizes using your actual queries and expected answers. Run it during development to find the optimal size for your specific content. The overlap ratio of 10% prevents boundary information loss without excessive chunk count inflation.
Using a single chunk size for all document types. A 500-character chunk works for technical documentation but is far too small for legal contracts with long clauses and too large for FAQ entries.
Tune chunk size per document type. Use smaller chunks (200-400 chars) for dense, fact-heavy content like FAQs and API docs. Use larger chunks (800-1500 chars) for narrative documents like reports and contracts where context spans longer passages.
Setting overlap to zero, which means information at chunk boundaries is split across two chunks and may not be retrievable by either embedding.
Use 10-20% overlap (e.g., 50-100 chars for a 500-char chunk). The overlap ensures boundary content is captured in at least one chunk with enough surrounding context for the embedding to represent it accurately.
Chunking by character count when the embedding model has a token limit. Characters and tokens do not have a fixed ratio -- 500 characters can be anywhere from 100 to 200 tokens depending on the text.
Use token-aware chunking with tiktoken or the tokenizer for your embedding model. Set the chunk size in tokens, not characters, to ensure no chunk exceeds the model's maximum input length.
Splitting in the middle of sentences, code blocks, or tables, producing chunks that are incomplete and whose embeddings do not represent any coherent meaning.
Use recursive splitting with separators that respect document structure: paragraph breaks, then sentence boundaries, then word boundaries. For code and markdown, use format-aware splitters that recognize code fences and table syntax.
Never evaluating chunking quality and treating chunk size as a set-and-forget configuration. As your document corpus grows and diversifies, the original settings may degrade retrieval performance.
Build a test set of queries with known correct source chunks. Measure retrieval recall at different chunk sizes and overlap values. Re-run this evaluation whenever you add a new document type to your knowledge base.
Document chunking is the most underrated optimization in RAG pipelines. The right chunk size and splitting strategy determine whether retrieval returns the precise passage that answers the query or a vague fragment that confuses the LLM. Start with recursive character splitting at 500 characters with 10% overlap. Graduate to token-aware or semantic chunking when your evaluation shows the baseline is not enough. Always benchmark on your own queries and documents -- the optimal chunk size depends on your content type, embedding model, and retrieval requirements.
Interactive lessons and challenges on Stanza, practice in VS Code, Cursor, or the web.
Interactive lessons and challenges, right in your code editor.
Check the free courses. No credit card.