RAG Integration & Prompt Engineering: Using Retrieved Context Effectively¶

Overview¶

RAG Integration combines retrieval with LLM generation in a cohesive pipeline. Prompt Engineering crafts prompts that effectively use retrieved documents. Both are critical for RAG quality.

Prompt structure: System + Context + Query + Examples
Context injection: How to include retrieved documents
Prompt patterns: Different templates for different use cases
Chain-of-thought: Improve reasoning with retrieved facts

RAG Pipeline¶

Full Pipeline¶

- ┌──────────────────────────────────────────────────────────┐
    - 1. User Query                                            │
    - "How do I fix a leaky roof?"                          │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - 2. Embed & Retrieve                                      │
    - - Convert query to embedding                          │
    - - Search vector database                             │
    - - Get top-5 documents                                │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - 3. (Optional) Rerank                                     │
    - - Score top-5 with cross-encoder                     │
    - - Reorder if needed                                  │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - 4. Construct Prompt                                      │
    - - System prompt + retrieved docs + query             │
    - - Inject context into prompt                         │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - 5. Generate Response                                     │
    - - LLM reads prompt + context                         │
    - - Generates grounded response                        │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - 6. Post-process                                          │
    - - Extract answer                                     │
    - - Format response                                    │
    - - Add citations                                      │
  - ┬─────────────────────────────────────────┘
                 ↓
- ┌──────────────────────────────────────────────────────────┐
    - Response with Citations                                  │
    - "To fix a leaky roof, you should...                     │
    - [1] Check the gutters (per document 1)                 │
    - [2] Apply sealant (per document 3)"                    │
  - ┘

Prompt Templates¶

Template 1: "Stuff" Pattern (Simple)¶

Concatenate all documents into prompt

Template:
"""
System: You are a helpful assistant.

Context:
{retrieved_documents}

Question: {user_query}

Answer: """

Example prompt:
"""
System: You are a helpful assistant.

Context:
Document 1: "Roof leaks often occur at flashings where the roof meets walls."
Document 2: "Check gutters for debris which can cause water backup."
Document 3: "Apply silicone sealant to small cracks before they expand."

Question: How do I fix a leaky roof?

Answer: """

Pros:
✅ Simple, easy to understand
✅ Works for small contexts

Cons:
❌ Limited context window
❌ All documents treated equally
❌ Loses document structure

Template 2: "Refine" Pattern (Quality)¶

Iteratively refine answer with multiple documents

Template:
"""
Based on the documents provided, answer the question.
Cite the specific document for each claim.

Document 1: {doc1_content}

Based on Document 1, here's what I know:
{intermediate_answer_1}

Document 2: {doc2_content}

Based on Documents 1 and 2:
{intermediate_answer_2}

Final answer: {final_answer}
"""

Benefit:
  - Chain-of-thought helps reasoning
  - Better integration of multiple documents

Template 3: "Map-Reduce" Pattern (Long Contexts)¶

Summarize each document, then synthesize

Algorithm:
1. For each document:
   - Prompt: "Summarize this document in 2 sentences related to: {query}"
   - Get summary

2. Combine summaries:
   - Prompt: "Based on these summaries, answer: {query}"
   - Get final answer

Example:

Question: "What are the different roof repair methods?"

Step 1 (Map):
Doc 1: "Shingles can be replaced by removing nails and installing new ones."
Doc 2: "Flat roofs need membrane repair using adhesive strips."
Doc 3: "Metal roofs can be patched with metal plates and caulk."

Step 2 (Reduce):
Prompt: "Based on: [summary1], [summary2], [summary3], answer: {query}"
Answer: "There are three main methods: shingle replacement, membrane repair, and metal patching."

Benefits:
✅ Handles many documents
✅ Parallelizable (summarize docs in parallel)
✅ Better for long context

Effective Prompt Patterns¶

Pattern 1: Chain-of-Thought with Context¶

Prompt:
"""
You are answering based on provided documents.

Documents:
{retrieved_documents}

Question: {query}

Let me think through this step by step:
1. First, I need to identify the relevant information:
2. From the documents, I find:
3. Now I can synthesize:

Answer: {let the model complete}
"""

Benefit:
  - Forces model to reason with documents
  - Better answers than direct generation

Pattern 2: Role-Based Context¶

Prompt:
"""
You are a roof repair expert. 
A homeowner asks you a question about roof repair.

You have access to these repair guides:
{retrieved_documents}

Homeowner: {user_query}

Expert: {generate response}
"""

Benefit:
  - Role-specific knowledge improves response

Pattern 3: Explicit Citation¶

Prompt:
"""
Answer the question using the provided documents.
For each claim, cite the source document.

Documents:
[Doc A] {content_a}
[Doc B] {content_b}

Question: {query}

Format your answer with citations:
- Claim 1 (Source: Doc A)
- Claim 2 (Source: Doc B)
"""

Benefit:
  - Model learns to cite sources
  - Improves transparency and verifiability

Common Mistakes¶

Mistake 1: Not Including Document Labels¶

Bad prompt:
"""
Context:
Lorem ipsum dolor sit amet...
Consectetur adipiscing elit...

Question: How does photosynthesis work?
"""

Problem:
  - Model doesn't know which doc is which
  - Can't cite sources
  - No control over which to prioritize

Good prompt:
"""
Context:
[Source 1 - Biology Textbook]:
Photosynthesis is the process where plants...

[Source 2 - Research Paper]:
Recent studies on photosynthesis show...

Question: How does photosynthesis work?
"""

Benefit:
  - Model knows document identity
  - Can reference by source
  - Can prioritize certain sources

Mistake 2: Not Limiting Context¶

Bad prompt:
"""
Context: {all_100_documents}

Question: {query}
"""

Problem:
  - Exceeds context window
  - Model confused by too much information
  - Quality drops (lost in noise)

Good approach:
```python
# Limit to top-5 most relevant
retrieved_docs = vector_search(query, k=100)
reranked_docs = rerank_cross_encoder(query, retrieved_docs[:100])
selected_docs = reranked_docs[:5]  # Use only top-5

prompt = construct_prompt(query, selected_docs)

Mistake 3: Ignoring Context Freshness¶

Bad approach:
"""
Context: {old_documents_from_2020}

Question: What are 2024 trends?
"""

Problem:
  - Model hallucinates based on old info
  - No recent data in context

Good approach:
```python
# Filter documents by recency
from datetime import datetime, timedelta

recent_docs = [d for d in docs if d['date'] > datetime.now() - timedelta(days=365)]
if not recent_docs:
    return "I don't have recent information on this topic."

prompt = construct_prompt(query, recent_docs)

---

## Optimization Techniques

### Prompt Optimization

```python
def optimize_prompt_for_retrieval(prompt_template):
    """
    Guidelines for RAG-friendly prompts:
    """

    # 1. Be explicit about using documents
    # Bad: "Answer the question"
    # Good: "Based on the provided documents, answer:"

    # 2. Specify citation format
    # Bad: "Answer question"
    # Good: "Answer and cite which document supports each claim"

    # 3. Set expectations
    # Bad: "What do you know about X?"
    # Good: "Based ONLY on the provided documents, what is X?"

    # 4. Include fallback
    # Bad: "Answer the question"
    # Good: "If not found in documents, say 'I don't have information on this'"

    # 5. Add examples
    prompt = """
    Example:
    Q: What is X?
    A: X is [fact from doc 1]. Additionally, [fact from doc 2].
    """

    return prompt

Temperature & Parameters¶

For RAG, adjust LLM parameters:

Factual questions:
  - temperature=0.1 (low, deterministic)
  - top_p=0.8
  - Reason: Single correct answer, need consistency

Complex reasoning:
  - temperature=0.5 (moderate)
  - top_p=0.9
  - Reason: Multiple valid answers, need some creativity

Summarization:
  - temperature=0.3
  - max_tokens=500
  - Reason: Concise, focused output

General RAG:
  - temperature=0.3-0.5 (balanced)
  - Reduce hallucinations while allowing reasoning

Advanced: Iterative RAG¶

Multi-Turn RAG¶

Instead of one retrieval, multiple rounds:

Round 1: Answer based on initial retrieval
  - "What is X?"
  - Retrieve docs, answer

Round 2: Refine answer based on follow-up
  - "Tell me more about X.1"
  - Retrieve docs specifically about X.1
  - Refine answer

Round 3: Validate answer
  - "Do X and Y contradict?"
  - Check documents for contradictions
  - Resolve or flag inconsistencies

Benefit:
  - Better multi-step reasoning
  - Can verify facts with new retrievals

Key Takeaways¶

📋 Prompt structure: System + Context + Query
📚 Stuff pattern: Simple, works for small contexts
🔄 Chain-of-thought: Improves reasoning with documents
📍 Citation: Include document labels and sources
🎯 Temperature: Lower (0.1-0.3) for factual RAG

Rag Fundamentals - Overview
Retrieval Strategies - What to retrieve
Chunking & Document Preparation - How to structure documents