Retrieval-Augmented Generation (RAG)

Overview

Retrieval-Augmented Generation (RAG) combines retrieval (finding relevant documents) with generation (creating responses). Instead of relying solely on model knowledge, RAG fetches relevant context before generating, dramatically improving accuracy and reducing hallucinations.

• Core Idea: Retrieve → Context → Generate
• Key Benefit: Reduce hallucinations, enable long-term memory, update knowledge without retraining
• Architecture: Retriever + LLM (two-stage system)
• Performance: 10-50% accuracy improvement typical
• Adoption: OpenAI's ChatGPT plugins, LangChain, industry standard for Q&A

---

The Problem RAG Solves

Hallucinations & Knowledge Limitations

LLM without RAG:

User: "What was the total revenue of Acme Corp in Q3 2024?"

Model response (hallucinated):
 - "Acme Corp's Q3 2024 revenue was $45.2 million"
 - Model never trained on 2024 data (knowledge cutoff: 2023)
 - Number is completely made up
 - User gets false information! 

Problem:
 - Model trained on fixed dataset (knowledge cutoff)
 - Can't access current information
 - Hallucinations fill gaps
 - Unreliable for: recent data, private documents, proprietary info

Knowledge Bottleneck

Scenario 1: User has 10,000 internal documents
 - Fine-tune model on all? (expensive, time-consuming)
 - Include in prompt? (context window limit, ~4K-128K tokens)
 - Store in model weights? (requires retraining)
 - Problem: Can't efficiently use external knowledge

Solution: Retrieve only relevant documents!
 - For each query, find relevant documents
 - Include only those in context
 - Model focuses on what's relevant
 - No retraining needed
 - Updates happen automatically as documents change!

---

How RAG Works

Architecture

- ┌──────────────────────────────────────────────────────────┐
 - User Query: "How does blood clotting work?" │
 - ┬─────────────────────────────────────┘
 ↓
- ┌────────────────────────┐
 - Stage 1: Retrieval │
 - ┤
 - Search knowledge base │
 - for relevant documents │
 - ┬───────────────┘
 ↓
- ┌────────────────────────────────────────┐
 - Retrieve top-K documents: │
 - - Wikipedia article on hemostasis │
 - - Medical journal on coagulation │
 - - Textbook chapter on clotting factors │
 - ┬───────────────────────────────┘
 ↓
- ┌────────────────────────────────────────────────────────────┐
 - Stage 2: Prompt Construction │
 - ┤
 - Prompt = [instruction] + [retrieved docs] + [query] │
│ │
 - Example: │
 - "Based on the following documents, answer the question." │
 - "[Document 1: hemostasis is the process...]" │
 - "[Document 2: coagulation cascade involves...]" │
 - "Question: How does blood clotting work?" │
 - ┬───────────────────────────────────────────────────┘
 ↓
- ┌────────────────────────────────────────────────────────────┐
 - Stage 3: Generation │
 - ┤
 - LLM reads documents and query │
 - Generates accurate response grounded in documents │
 - ┬───────────────────────────────────────────────────┘
 ↓
- ┌────────────────────────────────────────────────────────────┐
 - Generated Response (accurate, grounded): │
 - "Blood clotting is a process called hemostasis that │
 - involves three main stages: vascular response, │
 - platelet aggregation, and coagulation cascade..." │
 - ┘

Why RAG Works

Key insight: Models are good at reading, not memorization!

Standard LLM:
 - Knowledge in model weights
 - Fixed at training time
 - Must memorize everything
 - Limited by training data
 - Result: Hallucinations

RAG LLM:
 - Knowledge in external documents
 - Updated in real-time
 - Only reads what's relevant
 - No retraining needed
 - Result: Grounded, accurate responses

Analogy:
 - Standard LLM: Student with memorized textbook
 - RAG LLM: Student with access to textbook (can look up answers!)
 - RAG student answers better questions!

---

RAG vs Fine-tuning

Comparison

Approach Memory Speed Quality Freshness Cost Use Case
─────────────────────────────────────────────────────────────────
Fine-tune High Slow Best Stale $$$$ Permanent skills
RAG Medium Fast Good Real-time $ Dynamic knowledge
Hybrid High Med Excellent Real-time $$ Production
Few-shot Low Fast Fair Real-time $ Testing

Detailed comparison:

Fine-tuning:
Best quality (baked into model)
Fast inference (no retrieval)
Expensive training
Stale knowledge (can't update easily)
Fixed domain

RAG:
Real-time updates
No retraining
Flexible (works with any document)
Cheap ($100s instead of $10,000s)
Slightly lower quality (still excellent)
Requires retrieval step (slightly slower)

Hybrid (RAG + fine-tuning):
Best of both: skills + knowledge
Fast + fresh + accurate
Most complex
Higher cost than RAG alone
 - Typical production setup!

-

Core Components

1. Retriever

Purpose: Find relevant documents

Input: Query ("How does blood clotting work?")
 ↓
- ┌─────────────────────────────┐
 - Retriever (search engine) │
 - - BM25 (keyword search) │
 - - Dense (embedding search) │
 - - Hybrid (both) │
 - ┘
 ↓
Output: Top-K documents
 - Document 1 (relevance: 0.95)
 - Document 2 (relevance: 0.87)
 - Document 3 (relevance: 0.82)
 -...

2. Knowledge Base / Vector Database

Stores documents in searchable format

Traditional:
 - SQL database (keyword search only)
 - Slow for large datasets
 - Binary match (relevant or not)

Vector database:
 - Stores document embeddings
 - Fast semantic search
 - Relevance scores (0-1)
 - Can find similar concepts

Examples:
 - Pinecone (managed)
 - Weaviate (self-hosted)
 - Qdrant (self-hosted)
 - Milvus (self-hosted)
 - FAISS (library)

3. LLM (Generative Model)

Purpose: Read retrieved documents, generate response

Input:
 - System prompt: "You are a helpful assistant"
 - Retrieved documents: [context]
 - User query: "Question:..."

Process:
 - Read documents
 - Understand query
 - Synthesize response
 - Generate text

Key: Response is grounded in documents (less hallucination!)

---

RAG Pipeline Example

Code Flow

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Pinecone
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# Step 1
embeddings = OpenAIEmbeddings()
vectorstore = Pinecone.from_existing_index(
 index_name="medical-docs",
 embedding=embeddings
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Step 2
llm = OpenAI(model="gpt-3.5-turbo", temperature=0)

# Step 3
qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff", # Stuff retrieved docs into prompt
 retriever=retriever,
 return_source_documents=True
)

# Step 4
query = "How does blood clotting work?"
result = qa_chain({"query": query})

print(result["result"]) # Generated response
print(result["source_documents"]) # Which docs were used

-

When to Use RAG

Use RAG When

Knowledge is external (documents, databases)
Knowledge changes frequently (news, prices)
Knowledge is proprietary (internal docs)
Need to cite sources
Context window is limited
Can't afford retraining
Need real-time information

Use Fine-tuning When

Knowledge is stable (domain skills)
Need maximum quality on specific task
Knowledge should be in model (inference speed)
Cost/time not critical

Use Hybrid (RAG + Fine-tuning) When

Need both stable skills AND dynamic knowledge
Production quality is critical
Have budget for both approaches
Want best possible results

-

Common Misconceptions

"RAG replaces fine-tuning"
RAG and fine-tuning are complementary!
 - Fine-tune for stable knowledge/skills
 - RAG for dynamic/external knowledge

"More documents = better results"
Retrieval quality matters more than quantity
 - Retrieve top-3 relevant documents
 - Retrieve top-100 irrelevant documents ← worse!

"RAG is too slow"
Retrieval is fast (<100ms)
 - Vector search: ~10-50ms
 - LLM generation: ~1-5 seconds
 - Retrieval is <5% of latency!

"RAG doesn't need fine-tuning"
Hybrid approach (RAG + fine-tuning) is best
 - Fine-tune to understand domain-specific concepts
 - RAG to access actual data/documents

-

Key Metrics

RAG evaluation metrics:

1. Retrieval quality
 - Precision: % of retrieved docs are relevant
 - Recall: % of relevant docs were retrieved
 - MRR (Mean Reciprocal Rank): ranking quality

2. Generation quality
 - BLEU/ROUGE: Text similarity to reference
 - Factuality: % of statements are factually correct
 - Grounding: % of statements cite documents

3. End-to-end
 - Exact match: User's query perfectly answered
 - F1 score: Balanced metric
 - Human evaluation: Gold standard

---

Key Takeaways

RAG: Retrieve relevant documents, then generate responses Hallucination reduction: Ground responses in documents Real-time knowledge: Update documents without retraining Fast & cheap: Simpler than fine-tuning Hybrid best: RAG for knowledge + fine-tuning for skills

-

Related Notes in RAG Subdirectory

• [Vector Databases & Embeddings](/01-modeling/03-inference/03-knowledge-integration/rag/(vector-databases-embeddings/) - Storage and retrieval
• Retrieval Strategies - Different retrieval approaches
• [Chunking & Document Preparation](/01-modeling/03-inference/03-knowledge-integration/rag/(chunking-document-preparation/) - Data preprocessing
• [Reranking & Ranking](/01-modeling/03-inference/03-knowledge-integration/rag/(reranking-ranking/) - Improving retrieval quality
• [Rag Integration & Prompt Engineering](/01-modeling/03-inference/03-knowledge-integration/rag/(rag-integration-prompt-engineering/) - Using with LLMs
• [Rag Evaluation & Metrics](/01-modeling/03-inference/03-knowledge-integration/rag/(rag-evaluation-metrics/) - Measuring performance
• Scaling Rag Systems - Production considerations