8. Evaluating RAG: retrieval and generation separately

Split your metrics or you will tune the wrong stage.

Retrieval metrics (need a labelled set of question → relevant chunk ids): - `recall@k` — was a correct chunk in the top k? This is the ceiling on answer quality. - `MRR` / `nDCG` — how highly was it ranked?

Generation metrics (given retrieved context): - Faithfulness / groundedness — is every claim supported by the context? - Answer relevance — does it address the question? - Citation precision — do cited sources actually support the claim?

python
def evaluate(dataset):
    r = mean(any(cid in retrieve(q, k=8) for cid in gold) for q, gold in dataset)
    g = mean(judge_faithfulness(answer(q), context(q)) for q, _ in dataset)
    return {"recall@8": r, "faithfulness": g}

Rule of thumb: if `recall@k` is low, fix chunking, hybrid search, or reranking. If recall is high but answers are wrong, fix the generation prompt.

Gotcha: end-to-end "accuracy" alone hides which stage regressed after a deploy.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges