RAG Evaluation & Metrics: Measuring Performance¶

Overview¶

RAG Evaluation measures both retrieval quality and generation quality. Metrics quantify performance at each stage. Critical for understanding and improving RAG systems.

Retrieval metrics: Precision, Recall, MRR, NDCG
Generation metrics: BLEU, ROUGE, F1, Factuality
End-to-end: Accuracy, Human evaluation
Practical: Automated + human evaluation

Evaluation Framework¶

Multi-Level Evaluation¶

Level 1: Retrieval Quality
  - Did we get relevant documents?
  - Metrics: Precision@K, Recall, MRR
  - Critical for RAG (garbage in = garbage out)

Level 2: Generation Quality
  - Did the LLM generate good response?
  - Metrics: BLEU, ROUGE, Factuality
  - Depends on both documents AND LLM

Level 3: End-to-End Quality
  - Does final answer solve user's problem?
  - Metrics: Exact match, F1, Human rating
  - What users care about most

Debugging:
  - If end-to-end is bad: Check retrieval + generation
  - If retrieval bad: Improve retriever/chunking/ranking
  - If generation bad: Improve prompt/LLM

Retrieval Metrics¶

Precision & Recall¶

Precision: Of documents we retrieved, how many are relevant?
  - Precision@5: Of top-5, how many relevant
  - Precision@10: Of top-10, how many relevant
  - Formula: (# relevant in top-K) / K

Recall: Of all relevant documents, how many did we find?
  - Formula: (# relevant found) / (total # relevant)
  - Typically computed on top-K retrieved

Example:
Query: "How to fix a door hinge?"
Total relevant documents: 5
Retrieved (top-10): 4 relevant

Precision@10 = 4 / 10 = 0.40 (40% of retrieved are relevant)
Recall = 4 / 5 = 0.80 (found 80% of relevant documents)

Trade-off:
  - High precision: Few but good results
  - High recall: Many results (more noise)
  - F1: Balanced metric = 2 * (precision * recall) / (precision + recall)

Targets:
  - Precision@5: >0.80
  - Precision@10: >0.70
  - Recall@100: >0.90

Mean Reciprocal Rank (MRR)¶

Metric: Average position of first relevant document

Formula: MRR = (1/N) * Σ (1 / rank_of_first_relevant)

Example:
Query 1: First relevant at position 1 → 1/1 = 1.0
Query 2: First relevant at position 3 → 1/3 = 0.33
Query 3: First relevant at position 10 → 1/10 = 0.1
MRR = (1.0 + 0.33 + 0.1) / 3 = 0.48

Interpretation:
  - MRR = 1.0: Perfect (always #1)
  - MRR = 0.5: Good (first relevant at ~2)
  - MRR = 0.1: Poor (first relevant at ~10)
  - Target: >0.7 (first relevant usually in top-3)

NDCG (Normalized Discounted Cumulative Gain)¶

Metric: Combines relevance + ranking position

Formula: 
NDCG = DCG / IDCG
where DCG = Σ (relevance_i / log2(rank_i + 1))

Example (graded relevance 0-3):
Retrieved ranking:
Rank 1: Relevance 3 → 3 / log2(2) = 3.0
Rank 2: Relevance 2 → 2 / log2(3) = 1.26
Rank 3: Relevance 1 → 1 / log2(4) = 0.5
Rank 4: Relevance 0 → 0
DCG = 3.0 + 1.26 + 0.5 = 4.76

Ideal ranking:
Rank 1: Relevance 3 → 3.0
Rank 2: Relevance 2 → 1.26
Rank 3: Relevance 1 → 0.5
Rank 4: Relevance 0 → 0
IDCG = 4.76

NDCG = 4.76 / 4.76 = 1.0 (perfect!)

Interpretation:
  - NDCG@10: Measure quality of top-10
  - Values: 0-1 (higher is better)
  - Target: >0.8

Benefit over Precision/Recall:
  - NDCG rewards relevant items at top
  - Accounts for ranking quality

Generation Metrics¶

BLEU Score¶

Metric: Overlap between generated and reference text

Formula: BLEU = BP * exp(Σ log(p_n) / N)
where p_n = precision of n-grams

Example:
Reference: "The cat sat on the mat"
Generated: "A cat sat on a mat"

1-gram matches: 4/6 = 0.67
2-gram matches: 3/5 = 0.60
BLEU ≈ 0.63

Range: 0-1 (1 = perfect match)

Interpretation:
  - BLEU > 0.4: Acceptable
  - BLEU > 0.6: Good
  - BLEU > 0.8: Excellent

Pros:
✅ Fast to compute
✅ No human effort

Cons:
❌ Penalizes paraphrases (same meaning, different words)
❌ Not always correlated with human judgment
❌ Bad for open-ended responses

ROUGE Score¶

Metric: Recall-based (emphasizes coverage)

Formula: ROUGE-N = Σ (n-gram matches) / Σ (n-grams in reference)

Similar to BLEU but recall-focused

Often better than BLEU for:
  - Summarization
  - Paraphrasing
  - Open-ended generation

Factuality / Correctness¶

Problem: BLEU/ROUGE don't measure factuality

Example:
Reference: "Einstein discovered relativity in 1905"
Generated: "Newton discovered relativity in 1905"

BLEU: 0.8 (high, most words match)
Factuality: 0 (completely wrong - Newton didn't discover relativity!)

Solution: Evaluate factuality separately

Manual evaluation:
  - Human checks if facts are correct

Automated evaluation:
  - Fact extraction + verification
  - Question generation + VQA
  - NLI (Natural Language Inference)
  - Check if generated text is entailed by context

Implementation:
```python
def evaluate_factuality(context, generated_text):
    """Check if generated text is factually consistent with context"""

    # Use NLI model
    from transformers import pipeline

    nli = pipeline("zero-shot-classification", 
                  model="facebook/bart-large-mnli")

    # Check entailment
    result = nli(generated_text, context)

    if result['labels'][0] == 'entailment':
        return 1.0  # Factually consistent
    elif result['labels'][0] == 'contradiction':
        return 0.0  # Factually contradicts
    else:
        return 0.5  # Neutral

Target: - >0.95 (almost no hallucinations)

---

## End-to-End Metrics

### Exact Match (EM)

Metric: Is the answer exactly correct?

Example: Question: "Who invented the light bulb?" Reference: "Thomas Edison" Generated: "Thomas Edison" EM: 1.0 (perfect match)

Generated: "Edison" EM: 0.0 (not exact, even though correct)

Range: 0-1

Use case: - Factoid questions (dates, names, places)

### F1 Score

Metric: Overlap between answer and reference

Formula: F1 = 2 * (Precision * Recall) / (Precision + Recall)

Less strict than EM, allows partial credit

Example: Question: "What are the three steps of photosynthesis?" Reference: "Light-dependent reactions, Calvin cycle, electron transport" Generated: "Light reactions and Calvin cycle"

EM: 0.0 (missing one) F1: 0.67 (gets 2/3 components)

Target: - F1 > 0.5: Acceptable - F1 > 0.7: Good - F1 > 0.8: Excellent

### Human Evaluation

Metrics rated by humans (gold standard):

Relevance (0-3)
3: Directly answers question
2: Partially addresses
1: Tangentially related
0: Unrelated
Factuality (0-1)
1: No hallucinations
0: Contains false information
Completeness (0-3)
3: Fully answers
2: Partially answers
1: Minimal information
0: Doesn't answer
Clarity (0-3)
3: Very clear
2: Reasonably clear
1: Hard to understand
0: Incomprehensible

Overall: Weighted average of metrics

Typical setup: - Rate 100-200 examples - Multiple raters (3-5) - Calculate inter-rater agreement (kappa > 0.7) - Aggregate ratings

---

## Evaluation Workflow

### Complete Evaluation Pipeline

```python
def evaluate_rag_system(queries, ground_truth, documents):
    """Comprehensive RAG evaluation"""

    results = {
        'retrieval': {},
        'generation': {},
        'end_to_end': {}
    }

    for query, ground_truth_answer in zip(queries, ground_truth):
        # Step 1: Retrieval evaluation
        retrieved_docs = retrieve(query, documents)

        precision_at_5 = calculate_precision(retrieved_docs[:5], query)
        recall = calculate_recall(retrieved_docs, query)
        mrr = calculate_mrr(retrieved_docs, query)

        results['retrieval']['precision@5'].append(precision_at_5)
        results['retrieval']['recall'].append(recall)
        results['retrieval']['mrr'].append(mrr)

        # Step 2: Generation evaluation
        generated_answer = generate_with_rag(query, retrieved_docs)

        bleu = calculate_bleu(generated_answer, ground_truth_answer)
        rouge = calculate_rouge(generated_answer, ground_truth_answer)
        factuality = evaluate_factuality(retrieved_docs, generated_answer)

        results['generation']['bleu'].append(bleu)
        results['generation']['rouge'].append(rouge)
        results['generation']['factuality'].append(factuality)

        # Step 3: End-to-end evaluation
        em = (generated_answer == ground_truth_answer)
        f1 = calculate_f1(generated_answer, ground_truth_answer)

        results['end_to_end']['em'].append(em)
        results['end_to_end']['f1'].append(f1)

    # Aggregate results
    for category in results:
        for metric in results[category]:
            avg = sum(results[category][metric]) / len(results[category][metric])
            print(f"{category}/{metric}: {avg:.3f}")

    return results

Benchmarking¶

Common RAG Benchmarks¶

Benchmark      Domain           # Questions  Task
──────────────────────────────────────────────────────
MS MARCO       Web search       100K+        Ranking
Natural QA     Wikipedia        200K         QA
SQuAD 2.0      Wikipedia        100K         Extractive QA
HotpotQA       Wikipedia        100K         Multi-hop QA
TREC DL        Web search       50K+         Ranking

Using benchmarks:
  - Evaluate your RAG system on public benchmarks
  - Compare against baselines
  - Track progress over time

Key Takeaways¶

📊 Evaluate retrieval AND generation separately
🎯 Precision@K: Crucial for RAG quality
📈 NDCG: Better than precision/recall for ranking
✅ Factuality: Critical metric (avoid hallucinations)
👤 Human evaluation: Gold standard for end-to-end

Rag Fundamentals - What to evaluate
Retrieval Strategies - What affects retrieval metrics
Rag Integration & Prompt Engineering - What affects generation metrics