The LangSmith RAG evaluation guide separates document relevance, answer correctness, faithfulness, and helpfulness. That split prevents one good score from hiding a broken stage.
A final answer score tells you that RAG failed. Separate retrieval and generation scores tell you what to fix.
What should a RAG evaluation test?
Test the two stages independently. Retrieval evaluation asks whether RAG found documents that are relevant to the question and whether the required evidence appeared within the retrieved set. Generation evaluation asks whether the response is correct, useful, and faithful to those documents. Retrieval and generation need separate scores because either stage can conceal the failure of the other.
A correct answer produced from irrelevant evidence may reflect memorized model knowledge rather than grounding. Relevant documents paired with an unsupported answer indicate a generation or prompt problem. Both cases can look acceptable if a team reviews only the final wording.
Which RAG metrics matter?
Retrieval recall measures whether the needed source appeared in the candidate set. Precision or relevance checks how much retrieved material was useful. Answer correctness compares the response with a reference answer when one exists. Faithfulness checks whether the answer’s claims follow from the retrieved context, while helpfulness checks whether it actually addresses the user’s task. No single metric represents end-to-end reliability.
Citation accuracy adds another boundary: a response may be factually correct but point to a source that does not support the cited claim. Tenant and permission checks should remain deterministic tests, not judge scores.
How do teams turn RAG failures into improvements?
Store representative and failed queries in an AI evaluation harness with the expected evidence and, where possible, a reference answer. Change one part of the pipeline, such as chunking, reranking, prompt, or model, then rerun the same set. A useful evaluation connects each weak score to a component the team can change.
An LLM as a judge can scale relevance, faithfulness, and helpfulness grading after calibration against human labels. Code checks remain better for exact constraints such as citation format, access rules, and source presence.
How is agentic RAG evaluated differently?
Agentic RAG can rewrite queries, choose among retrieval tools, inspect results, and retry. The final answer therefore depends on a trajectory rather than one retrieval call. Evaluation must inspect whether each decision improved the evidence or merely added latency and cost.
Score the trajectory as well as the answer: which query was issued, which source was selected, why the agent retried, and when it stopped. A grounded final response can still come from an unstable process that succeeds only after wasteful or unsafe steps.