Your RAG pipeline needs a test set

A retrieval system can sound convincing while missing the document that contains the answer. This notebook evaluates retrieval separately from generation using three synthetic questions with explicit relevance labels. The dataset makes each score easy to inspect.

Define what success means #

For each question, we record relevant document identifiers. Recall at $k$ measures how many relevant documents appear in the first $k$ results:

$\operatorname{Recall@k} = \frac{|R_k \cap G|}{|G|}$

Here $R_k$ is the retrieved set and $G$ is the ground-truth set. Answer faithfulness needs a separate evaluation against the retrieved evidence.

examples = [
    {"question": "How do I roll back?", "relevant": {"deploy", "recovery"}, "retrieved": ["deploy", "faq", "recovery"]},
    {"question": "Who owns alerts?", "relevant": {"oncall"}, "retrieved": ["oncall", "logs", "faq"]},
    {"question": "Where are backups?", "relevant": {"storage"}, "retrieved": ["faq", "logs", "deploy"]},
]
 
def recall_at_k(retrieved, relevant, k):
    if not relevant:
        raise ValueError("An evaluation example needs relevant documents")
    return len(set(retrieved[:k]) & relevant) / len(relevant)
 
scores = [recall_at_k(e["retrieved"], e["relevant"], 3) for e in examples]
print(scores)
print(f"Mean recall@3: {sum(scores) / len(scores):.3f}")

[1.0, 1.0, 0.0]
Mean recall@3: 0.667

Inspect the failure, not just the score #

The aggregate gives us a starting point. The third question tells us what to investigate: the relevant document was never retrieved. A better answer prompt cannot recover missing evidence.

rows = [(e["question"], score) for e, score in zip(examples, scores)]
rows

Keep the evaluation set honest #

Build coverage from representative queries and human relevance judgments. Include common tasks, rare terminology, and questions whose answers span multiple documents. Split development and held-out sets before repeatedly tuning a retriever.

Treat unanswered questions as a separate category. An empty ground-truth set is not a retrieval score of zero; it asks whether the system knows when to abstain.

Next experiment #

Hold the dataset fixed and compare chunk boundaries. Then inspect per-question changes before interpreting the average. Record the corpus version, tokenizer, embedding model, and retrieval settings beside every run.