Enterprise RAG on LangChain: fix retrieval first

Your enterprise RAG system will not fail where you are looking. The model takes the blame for a wrong answer, but the highest-leverage fixes sit a step earlier, in the chunks retrieval never returned. Fix retrieval first: contextual chunking, hybrid search and a reranker cut failed retrievals by 67% in Anthropic’s published evaluation, more than any model swap will buy you. LangChain ships every block of that stack, and LangSmith closes the loop with evaluators that catch the decay a demo never shows. This post walks the stack in the order the data says to build it: architecture, chunking, hybrid retrieval, reranking, then the eval loop that keeps it honest.

Retrieval fails long before the model does

The case for RAG has not changed: models have what the LangChain retrieval docs call finite context and static knowledge, so the system fetches relevant external knowledge at query time. What has changed is where production teams lose. When a RAG answer is wrong, read the retrieved set before blaming the model: in our reviews the passage that held the answer was split badly, embedded without its surroundings, outranked by a near-duplicate, or never indexed at all, and the model summarized the wrong context faithfully. Each of those is a retrieval defect, and each has a named fix that costs less than a frontier model upgrade.

That ordering is the practical point. Teams tune prompts and swap models because those changes take an afternoon, while the retrieval stack looks like plumbing. The plumbing is the product. In production RAG, the generating model is rarely the bottleneck; the chunks you hand it are.

Pick the architecture by who controls the loop

The current LangChain docs organize RAG into three architectures, and the choice comes down to control. In 2-Step RAG, retrieval always runs before generation, so latency is predictable and the number of LLM calls is capped. In agentic RAG, an agent decides when and how to retrieve during reasoning; the docs note that the only thing an agent needs for RAG behavior is access to tools that fetch external knowledge. Hybrid designs sit between, wrapping retrieval in validation steps: query enhancement, retrieval validation that can re-query when documents come back thin, and answer validation against the sources.

Start with 2-Step unless you can say which question class it fails. We made the same argument about agent control flow in LangGraph agents in production: give the model the smallest loop that answers the question, because every degree of freedom you grant is a failure mode you now have to evaluate.

Chunking is a retrieval decision, not preprocessing

Chunking looks like preprocessing and behaves like ranking: whatever boundaries you cut are the only retrieval units that exist. The splitters documentation is unusually direct about the default: “For most use cases, start with the RecursiveCharacterTextSplitter. It provides a solid balance between keeping context intact and managing chunk size.” It keeps paragraphs intact where it can and descends to sentences and words only when a unit exceeds the chunk size, and for HTML, Markdown or JSON the structure-based splitters follow the document’s own boundaries instead. LangChain’s knowledge base tutorial pins the working defaults:

from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    add_start_index=True,
)
chunks = splitter.split_documents(pages)
# the tutorial's 107-page PDF becomes 516 chunks

add_start_index=True keeps each chunk’s character offset, which you will want the first time a user asks where an answer came from. The deeper problem is that a chunk cut from its document loses its referents, and that is what contextual retrieval repairs: Anthropic’s published method prepends a short generated context to each chunk before embedding, and contextual embeddings alone reduced the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%, at a one-time cost of $1.02 per million document tokens. The same post names the escape hatch worth checking before you build anything: “If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods.”

Hybrid search wins the queries embeddings miss

Dense semantic search over a vector database matches meaning and shrugs at exact strings, which is a problem the first time a user searches for an error code, a part number or a clause reference. The fix is old technology kept on purpose: pair the dense index with BM25, the sparse retrieval algorithm that still wins on exact identifiers, and fuse the result sets. Weaviate’s hybrid search docs state the pattern plainly: “Hybrid search combines the results of a vector search and a keyword (BM25F) search by fusing the two result sets. The fusion method and the relative weights are configurable.” Their alpha weight runs from 0, pure keyword, to 1, pure vector, and Relative Score Fusion has been the default fusion method since v1.24, so hybrid search has become a configuration decision.

Anthropic’s evaluation above measured the same combination: adding contextual BM25 to contextual embeddings took the failure rate from 3.7% to 2.9%, a 49% total reduction from the 5.7% baseline. Scope it honestly, though: BM25 earns its keep on corpora full of identifiers and jargon, and metadata filtering on fields like date, product or tenant often cuts the candidate set more cheaply than either index.

Reranking is the cheapest accuracy left on the table

First-stage retrieval, dense or hybrid, is built for recall over a large corpus. A reranker is a second model that reorders a small candidate set by actual relevance to the query, and because it reads query and document together, it catches what bi-encoder embeddings structurally cannot. Cohere’s Rerank docs describe the endpoint simply: given a query and a list of documents, it indexes them from most to least semantically relevant, and the model lineup now runs to Rerank 5. The gains are published and specific: Cohere’s Rerank 3.5 announcement measured it +23.4% better than hybrid search and +30.8% better than BM25 on a financial services dataset, and Anthropic’s pipeline retrieved the top 150 candidates and passed the top 20 after reranking, closing the failure-rate ladder at 1.9%, the 67% reduction the intro promised.

The cost is one extra model call per query over a few dozen candidates, plus latency you should measure before you commit. If your error budget has room for exactly one retrieval upgrade this quarter, this is the one with published numbers behind it.

The eval loop is what catches silent decay

A RAG system passes its launch review and then the corpus moves: documents get rewritten, products get renamed, yesterday’s precedents contradict today’s policy. A RAG system decays through its data, not its code, so no deploy-time test will catch it decaying. The LangSmith RAG evaluation tutorial decomposes quality into four comparisons instead of one score: correctness (response vs reference answer), relevance (response vs input), groundedness (response vs retrieved docs), and retrieval relevance (retrieved docs vs input). The split is what makes the loop diagnostic. A grounded but irrelevant answer is a retrieval problem; a relevant but ungrounded one is the model freelancing, the failure we took apart in your agent’s summary is not a source.

Offline datasets cover what you thought to test. Production needs online evaluation: LangSmith’s online evaluations run LLM-as-a-judge evaluators against live traces, “to identify issues, measure improvements, and ensure consistent quality over time”, with a sampling rate for cost control, 0.1 to judge a tenth of traces. LangChain’s January 2026 newsletter states the operating principle we would underline twice: “Production traces become living test cases that shape what you evaluate next”. Wire groundedness and retrieval relevance as online judges and the silent decay stops being silent. Judge calibration is its own discipline, which we covered in LangSmith LLM-as-judge.

What we check before an agentic RAG build ships

The checklist we run before any agentic RAG evaluation gate comes from a production architecture review we did of a knowledge-capture system, one that retrieves rules and past human judgments and writes every correction back as a future precedent. Three checks from it transfer to almost every enterprise RAG build, and they are part of the LangChain implementation work we get hired for.

The trap: a correction captured into the store is not a correction the system will ever retrieve. Whether it resurfaces depends on the relevance threshold and top-N, so a good fix can be saved and never seen again, with no failing test.

First, the write-read round trip: after a human correction is written back, run a near-identical input and assert the new precedent actually appears in the retrieved set. Second, re-run the golden-set regression when the data changes, not only when the code deploys; retrieval breaks silently on a corpus update, which is exactly when nobody is watching CI. Third, record the embedding model version on every record from day one, because models get deprecated, similarity across embedding versions is meaningless, and re-embedding thousands of records is a migration you want planned rather than discovered. None of these require new tooling; all three are assertions your existing test harness can hold.

The takeaway

Enterprise RAG on LangChain is won in the retrieval stack. Choose the smallest architecture that answers your question class, take the documented chunking defaults and add context to the chunks, fuse BM25 with your vector index, and spend your one upgrade on a reranker: that chain, measured end to end in Anthropic’s published evaluation, removes two thirds of retrieval failures before the generating model sees a token. Then treat evaluation as the production half of the system, with online judges for groundedness and retrieval relevance, a round-trip test on every write path, and regressions that fire on data changes. The model will keep improving without you. The retrieval stack is the part of your system that only gets better because you built it that way, the same bet we unpacked in MCP is now built into LangChain.

Sources

• Retrieval, LangChain docs
• Text splitters, LangChain docs
• Build a knowledge base, LangChain docs
• Introducing Contextual Retrieval, Anthropic
• Hybrid search, Weaviate docs
• Rerank overview, Cohere docs
• Introducing Rerank 3.5, Cohere
• Evaluate a RAG application, LangSmith docs
• Online evaluations, LangSmith docs
• January 2026 LangChain newsletter, LangChain blog