What is recall at k in RAG retrieval?

Recall at k measures whether the known relevant evidence appears somewhere in the first k retrieved results for a query. It answers a simple question: did the retriever make the needed information available to the generator? It does not say whether the evidence was ranked first, nor whether the final answer used it correctly.

How is mean reciprocal rank different from recall at k?

Recall at k treats any relevant result within the cutoff as success. Mean reciprocal rank cares where the first relevant result appears, giving more credit when it is near the top. This is important because RAG prompts usually include only a limited number of chunks, and earlier chunks often influence the model more.

Why do groundedness checks matter if retrieval metrics are good?

Good retrieval only means the supporting material was fetched. The generator can still invent details, ignore retrieved evidence, overstate what a source says, or combine facts from conflicting chunks. Groundedness checks compare the final answer against the retrieved context, so they test whether the answer is actually supported rather than merely fluent.

What should I inspect when a RAG evaluation fails?

Look at the individual query, expected evidence, retrieved chunks, their ranks, filters applied, and final answer. Label the failure as missing evidence, poor ranking, weak chunk boundaries, insufficient context, excluded metadata, conflicting sources, or generator misuse. The fix depends on that label, not on the aggregate score alone.

RAG Evaluation and Retrieval Quality

RAG evaluation and retrieval quality is the measurement of whether a retrieval-augmented generation system finds the evidence needed to answer a query, ranks it usefully, and produces answers supported by that evidence. It separates retriever failures from generator failures using metrics, groundedness checks, and inspection of concrete bad examples.

RAG systems fail in two different places: the retriever may not fetch the evidence, or the generator may misuse evidence it was given. Treating the whole pipeline as one black box hides the cause. A plausible answer can be unsupported, while a bad answer can come from excellent retrieval. Evaluation is needed because the model can only ground its answer in chunks that are present, visible, and ranked high enough to matter.

A practical evaluation set contains representative questions, expected answers, and the source chunks or documents that justify those answers. For each question, run the retriever and compare its returned chunks with the known supporting evidence. Recall at k asks whether relevant evidence appears within the first k results. Mean reciprocal rank rewards putting the first relevant result earlier, which matters when only the highest-ranked chunks reach the prompt.

Groundedness is a separate check on the generated answer: does the answer actually follow from the retrieved context? This catches cases where retrieval succeeded but the model hallucinated, overgeneralised, ignored a caveat, or merged incompatible facts. The common misunderstanding is to treat retrieval metrics as answer-quality metrics. They are not. Retrieval metrics tell you whether the evidence was available; groundedness and answer review tell you whether it was used correctly.

The trade-off is that good RAG evaluation requires labelled examples, source references, and repeated manual review. Metrics are useful, but aggregate scores can conceal the fix. If recall improves as k grows, ranking, chunking, filters, hybrid search, or reranking may be the issue. If recall stays poor, documents may be missing, embeddings may be weak, chunks may break meaning, or query language may not match the corpus.

Engineers meet this work when tuning vector search, choosing chunk sizes, adding metadata filters, introducing rerankers, testing hybrid search, or debugging user reports of wrong answers. The useful workflow is to log queries, retrieved chunks, ranks, prompts, and outputs, then label failures: no relevant chunk, relevant chunk too low, insufficient context, conflicting sources, bad filter, or generator ignored evidence. Those labels point directly to engineering changes.

Go deeper

Everything DataRAG Evaluation and Retrieval QualityEvaluate a RAG retriever using recall at k, mean reciprocal rank, groundedness checks, and qualitative failure analysis.Open the lesson

Common questions

What is recall at k in RAG retrieval?: Recall at k measures whether the known relevant evidence appears somewhere in the first k retrieved results for a query. It answers a simple question: did the retriever make the needed information available to the generator? It does not say whether the evidence was ranked first, nor whether the final answer used it correctly.
How is mean reciprocal rank different from recall at k?: Recall at k treats any relevant result within the cutoff as success. Mean reciprocal rank cares where the first relevant result appears, giving more credit when it is near the top. This is important because RAG prompts usually include only a limited number of chunks, and earlier chunks often influence the model more.
Why do groundedness checks matter if retrieval metrics are good?: Good retrieval only means the supporting material was fetched. The generator can still invent details, ignore retrieved evidence, overstate what a source says, or combine facts from conflicting chunks. Groundedness checks compare the final answer against the retrieved context, so they test whether the answer is actually supported rather than merely fluent.
What should I inspect when a RAG evaluation fails?: Look at the individual query, expected evidence, retrieved chunks, their ranks, filters applied, and final answer. Label the failure as missing evidence, poor ranking, weak chunk boundaries, insufficient context, excluded metadata, conflicting sources, or generator misuse. The fix depends on that label, not on the aggregate score alone.

Every term