Document chunking splits source material into smaller passages before those passages are embedded and indexed for retrieval. The chunk becomes the unit a RAG system can find and place into an LLM's context, so its boundaries strongly affect whether an answer receives complete and relevant evidence.
Microsoft’s RAG information retrieval guide treats chunking as part of preparing a searchable corpus. Headings, paragraphs, pages, and code functions can all provide meaningful boundaries.
How should documents be chunked?
Start with the document’s own structure and keep one coherent idea in each passage. A chunk should be small enough to retrieve precisely and large enough to stand on its own. Fixed token windows are simple, but they may split a definition from its exception or a heading from its body. Late chunking takes a different route by producing embeddings after the model has seen the surrounding document context.
How does chunking affect RAG quality?
Each chunk receives embeddings and usually metadata before entering a vector database. Weak boundaries make relevant facts hard to find or return incomplete evidence. Test chunking with the real questions your RAG system must answer, because a clean-looking index does not prove useful retrieval.
Frequently asked questions
What is the best chunk size for RAG?
There is no universal size. It depends on document structure, question type, embedding model, retrieval method, and how much context the answer needs.
Should chunks overlap?
Small overlaps can preserve context across arbitrary boundaries, but large overlaps create duplicates and waste retrieval and context capacity. Structure-aware boundaries are usually preferable.
No advertising or tracking cookies, and our visitor counts are anonymous. The Cal.com booking widget loads only if you allow it. Privacy Policy.
The page itself, anything our host sets to serve and secure it, and the anonymous visitor count. Always on, and none of it stores anything on your device.
The Cal.com booking widget. Left off, a booking link opens the booking page instead of a popup, so you can still book a call.