Anthropic’s Contextual Retrieval guide describes generating 50 to 100 tokens of chunk-specific context, then prepending it before creating both embeddings and a BM25 index. The technique is useful because ordinary chunks often lose the subject that was obvious only in the original document.
A chunk that says what it belongs to is easier to retrieve than the same words lifted out of their document.
Why do ordinary chunks lose meaning?
RAG systems split a source into chunks so they can retrieve a small relevant passage instead of inserting a whole document in the prompt. A chunk may then say “the program”, “this policy”, or “they” without naming the product, customer, or section that gives those words meaning. The same short phrase can appear in many documents.
Contextual retrieval asks a model to create a concise prefix that situates the chunk in its source. The prefix preserves the document’s subject and scope at the moment retrieval needs to rank it. It works alongside sensible document chunking, rather than fixing chunks that are already too large, too small, or cut across a required table or section.
How does the indexing flow work?
For each chunk, the system provides the full source document and the chunk to a model. The model writes a short context statement. The system prefixes that statement to the original text, then indexes the combined text with a sparse method such as BM25 and with embeddings. At query time it runs both searches and combines their rankings, which is a form of hybrid search.
The retrieval stage can still use reranking to check the leading candidates against the full query. Contextual retrieval improves the candidate pool. Reranking chooses more carefully within it. These are different jobs, and one does not make the other redundant.
What is the difference between contextual retrieval and late chunking?
Both address the same lost-context problem, but they act at different stages. Late chunking encodes the whole document before it pools representations for each chunk. Contextual retrieval writes extra text that gives an individual chunk its source meaning before indexing. Late chunking needs an encoder that can process the document in one context. Contextual retrieval needs a generation pass for every chunk.
Choose the method that improves retrieval on your actual corpus, not the one that sounds more advanced. A short, clean document may need neither. A large policy library with repeated headings and ambiguous references may benefit a great deal.
What can go wrong in production?
The generated prefix can hallucinate, leak a sensitive part of the source into an unrelated chunk, or add enough tokens to increase storage and retrieval cost. Treat the prefix as derived data with a source identifier and permission checks. Re-create it when the document changes. Then evaluate the complete RAG system with real questions, including questions that should return no answer. Better retrieval is only useful when the answer remains grounded in the source it retrieved.