09.07 · Concept
RAG Evaluation and Retrieval Quality
Evaluate a RAG retriever using recall at k, mean reciprocal rank, groundedness checks, and qualitative failure analysis.
RAG retrieval quality is measured by whether the right evidence is found, how high it appears, and whether the final answer stays supported by that evidence. Useful evaluation separates retriever errors from generator errors, then combines ranked metrics with inspection of failed queries to identify fixes.
What this lesson answers
- how to evaluate RAG retrieval quality
- recall at k versus mean reciprocal rank
- how to check groundedness in RAG
Notes
Evaluating a RAG system starts with separating retrieval quality from generation quality. The retriever’s job is to return chunks that contain the information needed to answer the user’s question. Recall at k measures whether at least one relevant chunk appears in the top k retrieved results, which is useful because the generator can only answer from evidence it sees. Mean reciprocal rank adds another dimension: it rewards systems that place the first relevant result near the top.
Common questions
- What does recall at k tell me in RAG evaluation?
- Recall at k tells you whether known relevant evidence appears within the retrieved results considered by the system. It is useful because the generator cannot use evidence it never receives. Low recall means the retriever, index, chunking, filters, or source coverage may be preventing the answer from reaching the prompt.
- Why use mean reciprocal rank as well as recall?
- Mean reciprocal rank rewards putting the first useful result near the top, not merely somewhere in the retrieved set. That matters because context space is limited and earlier chunks usually have more influence on the generated answer. A system can have acceptable recall while still ranking relevant evidence too low.
- How do groundedness checks differ from retriever metrics?
- Retriever metrics check whether the right supporting material was returned and ranked well. Groundedness checks inspect whether the generated answer is actually justified by that material. They catch cases where retrieval succeeded but the model hallucinated, ignored evidence, merged facts incorrectly, or answered beyond the supplied context.
Short definition: what is RAG Evaluation and Retrieval Quality?