RAG Evaluation & Metrics

Overview

RAG Evaluation measures both retrieval quality and generation quality. Metrics quantify performance at each stage. Critical for understanding and improving RAG systems.

• Retrieval metrics: Precision, Recall, MRR, NDCG
• Generation metrics: BLEU, ROUGE, F1, Factuality
• End-to-end: Accuracy, Human evaluation
• Practical: Automated + human evaluation

-

Evaluation Framework

Multi-Level Evaluation

Level 1: Retrieval Quality
 - Did we get relevant documents?
 - Metrics: Precision@K, Recall, MRR
 - Critical for RAG (garbage in = garbage out)

Level 2: Generation Quality
 - Did the LLM generate good response?
 - Metrics: BLEU, ROUGE, Factuality
 - Depends on both documents AND LLM

Level 3: End-to-End Quality
 - Does final answer solve user's problem?
 - Metrics: Exact match, F1, Human rating
 - What users care about most

Debugging:
 - If end-to-end is bad: Check retrieval + generation
 - If retrieval bad: Improve retriever/chunking/ranking
 - If generation bad: Improve prompt/LLM

-

Retrieval Metrics

Precision & Recall

Precision: Of documents we retrieved, how many are relevant?
 - Precision@5: Of top-5, how many relevant
 - Precision@10: Of top-10, how many relevant
 - Formula: (# relevant in top-K) / K

Recall: Of all relevant documents, how many did we find?
 - Formula: (# relevant found) / (total # relevant)
 - Typically computed on top-K retrieved

Example:
Query: "How to fix a door hinge?"
Total relevant documents: 5
Retrieved (top-10): 4 relevant

Precision@10 = 4 / 10 = 0.40 (40% of retrieved are relevant)
Recall = 4 / 5 = 0.80 (found 80% of relevant documents)

Trade-off:
 - High precision: Few but good results
 - High recall: Many results (more noise)
 - F1: Balanced metric = 2 * (precision * recall) / (precision + recall)

Targets:
 - Precision@5: >0.80
 - Precision@10: >0.70
 - Recall@100: >0.90

Mean Reciprocal Rank (MRR)

Metric: Average position of first relevant document

Formula: MRR = (1/N) * Σ (1 / rank_of_first_relevant)

Example:
Query 1: First relevant at position 1 → 1/1 = 1.0
Query 2: First relevant at position 3 → 1/3 = 0.33
Query 3: First relevant at position 10 → 1/10 = 0.1
MRR = (1.0 + 0.33 + 0.1) / 3 = 0.48

Interpretation:
 - MRR = 1.0: Perfect (always #1)
 - MRR = 0.5: Good (first relevant at ~2)
 - MRR = 0.1: Poor (first relevant at ~10)
 - Target: >0.7 (first relevant usually in top-3)

NDCG (Normalized Discounted Cumulative Gain)

Metric: Combines relevance + ranking position

Formula: 
NDCG = DCG / IDCG
where DCG = Σ (relevance_i / log2(rank_i + 1))

Example (graded relevance 0-3):
Retrieved ranking:
Rank 1: Relevance 3 → 3 / log2(2) = 3.0
Rank 2: Relevance 2 → 2 / log2(3) = 1.26
Rank 3: Relevance 1 → 1 / log2(4) = 0.5
Rank 4: Relevance 0 → 0
DCG = 3.0 + 1.26 + 0.5 = 4.76

Ideal ranking:
Rank 1: Relevance 3 → 3.0
Rank 2: Relevance 2 → 1.26
Rank 3: Relevance 1 → 0.5
Rank 4: Relevance 0 → 0
IDCG = 4.76

NDCG = 4.76 / 4.76 = 1.0 (perfect!)

Interpretation:
 - NDCG@10: Measure quality of top-10
 - Values: 0-1 (higher is better)
 - Target: >0.8

Benefit over Precision/Recall:
 - NDCG rewards relevant items at top
 - Accounts for ranking quality

-

Generation Metrics

BLEU Score

Metric: Overlap between generated and reference text

Formula: BLEU = BP * exp(Σ log(p_n) / N)
where p_n = precision of n-grams

Example:
Reference: "The cat sat on the mat"
Generated: "A cat sat on a mat"

1-gram matches: 4/6 = 0.67
2-gram matches: 3/5 = 0.60
BLEU ≈ 0.63

Range: 0-1 (1 = perfect match)

Interpretation:
 - BLEU > 0.4: Acceptable
 - BLEU > 0.6: Good
 - BLEU > 0.8: Excellent

Pros:
Fast to compute
No human effort

Cons:
Penalizes paraphrases (same meaning, different words)
Not always correlated with human judgment
Bad for open-ended responses

ROUGE Score

Metric: Recall-based (emphasizes coverage)

Formula: ROUGE-N = Σ (n-gram matches) / Σ (n-grams in reference)

Similar to BLEU but recall-focused

Often better than BLEU for:
 - Summarization
 - Paraphrasing
 - Open-ended generation

Factuality / Correctness

Problem: BLEU/ROUGE don't measure factuality

Example:
Reference: "Einstein discovered relativity in 1905"
Generated: "Newton discovered relativity in 1905"

BLEU: 0.8 (high, most words match)
Factuality: 0 (completely wrong - Newton didn't discover relativity!)

Solution: Evaluate factuality separately

Manual evaluation:
 - Human checks if facts are correct

Automated evaluation:
 - Fact extraction + verification
 - Question generation + VQA
 - NLI (Natural Language Inference)
 - Check if generated text is entailed by context

Implementation:
```python
def evaluate_factuality(context, generated_text):
 """Check if generated text is factually consistent with context"""

 # Use NLI model
 from transformers import pipeline

 nli = pipeline("zero-shot-classification", 
 model="facebook/bart-large-mnli")

 # Check entailment
 result = nli(generated_text, context)

 if result['labels'][0] == 'entailment':
 return 1.0 # Factually consistent
 elif result['labels'][0] == 'contradiction':
 return 0.0 # Factually contradicts
 else:
 return 0.5 # Neutral

Target:

• 0.95 (almost no hallucinations)

-

## End-to-End Metrics

### Exact Match (EM)

Metric: Is the answer exactly correct?

Example: Question: "Who invented the light bulb?" Reference: "Thomas Edison" Generated: "Thomas Edison" EM: 1.0 (perfect match)

Generated: "Edison" EM: 0.0 (not exact, even though correct)

Range: 0-1

Use case:

• Factoid questions (dates, names, places)

### F1 Score

Metric: Overlap between answer and reference

Formula: F1 = 2 (Precision Recall) / (Precision + Recall)

Less strict than EM, allows partial credit

Example: Question: "What are the three steps of photosynthesis?" Reference: "Light-dependent reactions, Calvin cycle, electron transport" Generated: "Light reactions and Calvin cycle"

EM: 0.0 (missing one) F1: 0.67 (gets 2/3 components)

Target:

• F1 > 0.5: Acceptable
• F1 > 0.7: Good
• F1 > 0.8: Excellent

### Human Evaluation

Metrics rated by humans (gold standard):

1. Relevance (0-3)

2. 3: Directly answers question

3. 2: Partially addresses
4. 1: Tangentially related

5. 0: Unrelated

6. Factuality (0-1)

7. 1: No hallucinations

8. 0: Contains false information

9. Completeness (0-3)

10. 3: Fully answers

11. 2: Partially answers
12. 1: Minimal information

13. 0: Doesn't answer

14. Clarity (0-3)

15. 3: Very clear

16. 2: Reasonably clear
17. 1: Hard to understand
18. 0: Incomprehensible

Overall: Weighted average of metrics

Typical setup:

• Rate 100-200 examples
• Multiple raters (3-5)
• Calculate inter-rater agreement (kappa > 0.7)
• Aggregate ratings

-

## Evaluation Workflow

### Complete Evaluation Pipeline

```python
def evaluate_rag_system(queries, ground_truth, documents):
 """Comprehensive RAG evaluation"""

 results = {
 'retrieval': {},
 'generation': {},
 'end_to_end': {}
 }

 for query, ground_truth_answer in zip(queries, ground_truth):
 # Step 1: Retrieval evaluation
 retrieved_docs = retrieve(query, documents)

 precision_at_5 = calculate_precision(retrieved_docs[:5], query)
 recall = calculate_recall(retrieved_docs, query)
 mrr = calculate_mrr(retrieved_docs, query)

 results['retrieval']['precision@5'].append(precision_at_5)
 results['retrieval']['recall'].append(recall)
 results['retrieval']['mrr'].append(mrr)

 # Step 2: Generation evaluation
 generated_answer = generate_with_rag(query, retrieved_docs)

 bleu = calculate_bleu(generated_answer, ground_truth_answer)
 rouge = calculate_rouge(generated_answer, ground_truth_answer)
 factuality = evaluate_factuality(retrieved_docs, generated_answer)

 results['generation']['bleu'].append(bleu)
 results['generation']['rouge'].append(rouge)
 results['generation']['factuality'].append(factuality)

 # Step 3: End-to-end evaluation
 em = (generated_answer == ground_truth_answer)
 f1 = calculate_f1(generated_answer, ground_truth_answer)

 results['end_to_end']['em'].append(em)
 results['end_to_end']['f1'].append(f1)

 # Aggregate results
 for category in results:
 for metric in results[category]:
 avg = sum(results[category][metric]) / len(results[category][metric])
 print(f"{category}/{metric}: {avg:.3f}")

 return results

---

Benchmarking

Common RAG Benchmarks

Benchmark Domain # Questions Task
──────────────────────────────────────────────────────
MS MARCO Web search 100K+ Ranking
Natural QA Wikipedia 200K QA
SQuAD 2.0 Wikipedia 100K Extractive QA
HotpotQA Wikipedia 100K Multi-hop QA
TREC DL Web search 50K+ Ranking

Using benchmarks:
 - Evaluate your RAG system on public benchmarks
 - Compare against baselines
 - Track progress over time

-

Key Takeaways

Evaluate retrieval AND generation separately Precision@K: Crucial for RAG quality NDCG: Better than precision/recall for ranking Factuality: Critical metric (avoid hallucinations) Human evaluation: Gold standard for end-to-end

-

Related Notes in RAG Subdirectory

• Rag Fundamentals - What to evaluate
• Retrieval Strategies - What affects retrieval metrics
• [Rag Integration & Prompt Engineering](/01-modeling/03-inference/03-knowledge-integration/rag/(rag-integration-prompt-engineering/) - What affects generation metrics