RAG Integration & Prompt Engineering¶
Overview¶
RAG Integration combines retrieval with LLM generation in a cohesive pipeline. Prompt Engineering crafts prompts that effectively use retrieved documents. Both are critical for RAG quality.
- Prompt structure: System + Context + Query + Examples
- Context injection: How to include retrieved documents
- Prompt patterns: Different templates for different use cases
- Chain-of-thought: Improve reasoning with retrieved facts
RAG Pipeline¶
Full Pipeline¶
- ┌──────────────────────────────────────────────────────────┐
- 1. User Query │
- "How do I fix a leaky roof?" │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- 2. Embed & Retrieve │
- - Convert query to embedding │
- - Search vector database │
- - Get top-5 documents │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- 3. (Optional) Rerank │
- - Score top-5 with cross-encoder │
- - Reorder if needed │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- 4. Construct Prompt │
- - System prompt + retrieved docs + query │
- - Inject context into prompt │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- 5. Generate Response │
- - LLM reads prompt + context │
- - Generates grounded response │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- 6. Post-process │
- - Extract answer │
- - Format response │
- - Add citations │
- ┬─────────────────────────────────────────┘
↓
- ┌──────────────────────────────────────────────────────────┐
- Response with Citations │
- "To fix a leaky roof, you should... │
- [1] Check the gutters (per document 1) │
- [2] Apply sealant (per document 3)" │
- ┘
Prompt Templates¶
Template 1: "Stuff" Pattern (Simple)¶
Concatenate all documents into prompt
Template:
"""
System: You are a helpful assistant.
Context:
{retrieved_documents}
Question: {user_query}
Answer: """
Example prompt:
"""
System: You are a helpful assistant.
Context:
Document 1: "Roof leaks often occur at flashings where the roof meets walls."
Document 2: "Check gutters for debris which can cause water backup."
Document 3: "Apply silicone sealant to small cracks before they expand."
Question: How do I fix a leaky roof?
Answer: """
Pros:
Simple, easy to understand
Works for small contexts
Cons:
Limited context window
All documents treated equally
Loses document structure
Template 2: "Refine" Pattern (Quality)¶
Iteratively refine answer with multiple documents
Template:
"""
Based on the documents provided, answer the question.
Cite the specific document for each claim.
Document 1: {doc1_content}
Based on Document 1, here's what I know:
{intermediate_answer_1}
Document 2: {doc2_content}
Based on Documents 1 and 2:
{intermediate_answer_2}
Final answer: {final_answer}
"""
Benefit:
- Chain-of-thought helps reasoning
- Better integration of multiple documents
Template 3: "Map-Reduce" Pattern (Long Contexts)¶
Summarize each document, then synthesize
Algorithm:
1. For each document:
- Prompt: "Summarize this document in 2 sentences related to: {query}"
- Get summary
2. Combine summaries:
- Prompt: "Based on these summaries, answer: {query}"
- Get final answer
Example:
Question: "What are the different roof repair methods?"
Step 1 (Map):
Doc 1: "Shingles can be replaced by removing nails and installing new ones."
Doc 2: "Flat roofs need membrane repair using adhesive strips."
Doc 3: "Metal roofs can be patched with metal plates and caulk."
Step 2 (Reduce):
Prompt: "Based on: [summary1], [summary2], [summary3], answer: {query}"
Answer: "There are three main methods: shingle replacement, membrane repair, and metal patching."
Benefits:
Handles many documents
Parallelizable (summarize docs in parallel)
Better for long context
Effective Prompt Patterns¶
Pattern 1: Chain-of-Thought with Context¶
Prompt:
"""
You are answering based on provided documents.
Documents:
{retrieved_documents}
Question: {query}
Let me think through this step by step:
1. First, I need to identify the relevant information:
2. From the documents, I find:
3. Now I can synthesize:
Answer: {let the model complete}
"""
Benefit:
- Forces model to reason with documents
- Better answers than direct generation
Pattern 2: Role-Based Context¶
Prompt:
"""
You are a roof repair expert.
A homeowner asks you a question about roof repair.
You have access to these repair guides:
{retrieved_documents}
Homeowner: {user_query}
Expert: {generate response}
"""
Benefit:
- Role-specific knowledge improves response
Pattern 3: Explicit Citation¶
Prompt:
"""
Answer the question using the provided documents.
For each claim, cite the source document.
Documents:
[Doc A] {content_a}
[Doc B] {content_b}
Question: {query}
Format your answer with citations:
- Claim 1 (Source: Doc A)
- Claim 2 (Source: Doc B)
"""
Benefit:
- Model learns to cite sources
- Improves transparency and verifiability
Common Mistakes¶
Mistake 1: Not Including Document Labels¶
Bad prompt:
"""
Context:
Lorem ipsum dolor sit amet...
Consectetur adipiscing elit...
Question: How does photosynthesis work?
"""
Problem:
- Model doesn't know which doc is which
- Can't cite sources
- No control over which to prioritize
Good prompt:
"""
Context:
[Source 1 - Biology Textbook]:
Photosynthesis is the process where plants...
[Source 2 - Research Paper]:
Recent studies on photosynthesis show...
Question: How does photosynthesis work?
"""
Benefit:
- Model knows document identity
- Can reference by source
- Can prioritize certain sources
Mistake 2: Not Limiting Context¶
Bad prompt:
"""
Context: {all_100_documents}
Question: {query}
"""
Problem:
- Exceeds context window
- Model confused by too much information
- Quality drops (lost in noise)
Good approach:
```python
# Limit to top-5 most relevant
retrieved_docs = vector_search(query, k=100)
reranked_docs = rerank_cross_encoder(query, retrieved_docs[:100])
selected_docs = reranked_docs[:5] # Use only top-5
prompt = construct_prompt(query, selected_docs)
Mistake 3: Ignoring Context Freshness¶
Bad approach:
"""
Context: {old_documents_from_2020}
Question: What are 2024 trends?
"""
Problem:
- Model hallucinates based on old info
- No recent data in context
Good approach:
```python
# Filter documents by recency
from datetime import datetime, timedelta
recent_docs = [d for d in docs if d['date'] > datetime.now() - timedelta(days=365)]
if not recent_docs:
return "I don't have recent information on this topic."
prompt = construct_prompt(query, recent_docs)
---
## Optimization Techniques
### Prompt Optimization
```python
def optimize_prompt_for_retrieval(prompt_template):
"""
Guidelines for RAG-friendly prompts:
"""
# 1. Be explicit about using documents
# Bad: "Answer the question"
# Good: "Based on the provided documents, answer:"
# 2. Specify citation format
# Bad: "Answer question"
# Good: "Answer and cite which document supports each claim"
# 3. Set expectations
# Bad: "What do you know about X?"
# Good: "Based ONLY on the provided documents, what is X?"
# 4. Include fallback
# Bad: "Answer the question"
# Good: "If not found in documents, say 'I don't have information on this'"
# 5. Add examples
prompt = """
Example:
Q: What is X?
A: X is [fact from doc 1]. Additionally, [fact from doc 2].
"""
return prompt
Temperature & Parameters¶
For RAG, adjust LLM parameters:
Factual questions:
- temperature=0.1 (low, deterministic)
- top_p=0.8
- Reason: Single correct answer, need consistency
Complex reasoning:
- temperature=0.5 (moderate)
- top_p=0.9
- Reason: Multiple valid answers, need some creativity
Summarization:
- temperature=0.3
- max_tokens=500
- Reason: Concise, focused output
General RAG:
- temperature=0.3-0.5 (balanced)
- Reduce hallucinations while allowing reasoning
Advanced: Iterative RAG¶
Multi-Turn RAG¶
Instead of one retrieval, multiple rounds:
Round 1: Answer based on initial retrieval
- "What is X?"
- Retrieve docs, answer
Round 2: Refine answer based on follow-up
- "Tell me more about X.1"
- Retrieve docs specifically about X.1
- Refine answer
Round 3: Validate answer
- "Do X and Y contradict?"
- Check documents for contradictions
- Resolve or flag inconsistencies
Benefit:
- Better multi-step reasoning
- Can verify facts with new retrievals
Key Takeaways¶
Prompt structure: System + Context + Query Stuff pattern: Simple, works for small contexts Chain-of-thought: Improves reasoning with documents Citation: Include document labels and sources Temperature: Lower (0.1-0.3) for factual RAG
-
Related Notes in RAG Subdirectory¶
- Rag Fundamentals - Overview
- Retrieval Strategies - What to retrieve
- [Chunking & Document Preparation](/01-modeling/03-inference/03-knowledge-integration/rag/(chunking-document-preparation/) - How to structure documents