Speculative Decoding: Complete Technical Guide

Overview

Speculative Decoding is an inference optimization technique that uses a smaller draft model to generate multiple predicted tokens ahead, then verifies them in parallel with a large target model. This breaks the sequential bottleneck of token generation, achieving 2-4x speedup without changing model outputs.

• Paper: "Faster Transformer Decoding through Non-Autoregressive Iterative Refinement"
• Key Innovation: Parallel verification of predicted tokens
• Impact: 2-4x speedup, especially for long-context and batch inference
• Complexity: Medium (algorithm is straightforward but careful implementation needed)
• Adoption: Growing in vLLM, SGLang, and other inference engines

---

The Problem: Sequential Token Generation Bottleneck

Why Token Generation is Slow

Standard Autoregressive Decoding (Sequential):

For generating N tokens:
  - Token 1: Forward pass → get output
  - Token 2: Forward pass → get output
  - Token 3: Forward pass → get output
  - ...
  - Token N: Forward pass → get output

Key observation:
  - Each token requires ONE forward pass
  - Passes cannot be parallelized (each depends on previous)
  - Must wait for each token to start next
  - LATENCY BOTTLENECK! (can't parallelize)

Example: Generate 256 tokens
Llama 2 7B inference:
  - Model size: 14GB (FP16)
  - Forward pass: 50ms (on A100)
  - Total time: 256 × 50ms = 12.8 seconds
  - But GPU is idle between passes!
  - Reason: Memory-bound operation + sequential dependency

Why it's memory-bound:
  - Load 14GB model weights
  - Process 1 token (tiny computation)
  - Memory bandwidth limits speed
  - Can't parallelize due to dependencies

Conclusion:
Each token takes ~50ms (not because of compute,
but because of memory bandwidth + sequential structure)

The Sequential Dependency Problem

Standard Autoregressive Generation:

Input: "The quick brown"

Step 1:
  - Input: [The, quick, brown]
  - Forward pass: Process all 3 tokens
  - Output: Prediction for token 4 = "fox"
  - Time: 50ms

Step 2:
  - Input: [The, quick, brown, fox]
  - Forward pass: Process all 4 tokens ← REDUNDANT!
  - Output: Prediction for token 5 = "jumps"
  - Time: 50ms

Step 3:
  - Input: [The, quick, brown, fox, jumps]
  - Forward pass: Process all 5 tokens ← MORE REDUNDANT!
  - Output: Prediction for token 6 = "over"
  - Time: 50ms

Observations:
1. Recompute tokens 1-3 again and again (with KV cache, stored)
2. Each step must wait for previous to complete
3. Only the LAST token's output matters
4. First N-1 tokens' predictions are ignored!
5. This is inherently sequential (can't parallelize)

With KV cache optimization:
  - Don't recompute, just reuse (good)
  - But still must generate ONE token per pass
  - Still sequential (bad)
  - Still 256 passes for 256 tokens

Performance Ceiling

Memory Bandwidth Analysis:

A100 GPU Memory Bandwidth: 2 TB/s
Llama 2 7B Model: 14GB weights (FP16)

Per token:
  - Load model: 14GB
  - Process: 1 token (minimal computation)
  - Effective tokens/sec: 2 TB/s / (14 × 10^9 bytes) = 142 tokens/sec
  - BUT with overhead: ~100-120 tokens/sec in practice
  - Time per token: 1/100 = 10ms

But we measure 50ms per token (5x worse!)
Why?
  - Not 100% memory bandwidth utilization
  - Other overheads (scheduling, I/O)
  - Kernel launch overhead
  - Result: ~50ms per token

For 256 tokens: 256 × 50ms = 12.8 seconds

Speculative Decoding changes this!

---

Speculative Decoding: The Solution

Core Idea: Predict Ahead, Verify in Parallel

Key Insight:
"Instead of generating 1 token per pass,
 generate K tokens speculatively,
 then verify all K in parallel!"

Standard:
Pass 1 → Token 1 → Pass 2 → Token 2 → Pass 3 → Token 3 ...
Total: N passes for N tokens ❌ (sequential)

Speculative:
Draft: Predict tokens 1-4 quickly
Verify: Check all 4 with large model in parallel ✓
Result: ~N/4 passes for N tokens (4x speedup!)

Architecture:
- ┌──────────────────────────────────┐
    - Input Prompt                     │
  - ┘
          ↓
- ┌──────────────────────────────────┐
    - Draft Model (small, fast)        │
    - Generate K=4 predicted tokens    │
  - ┤
    - Predictions:                     │
    - [fox, jumps, over, the]          │
  - ┘
          ↓
- ┌──────────────────────────────────────────────────┐
    - Target Model (large, slow)                      │
    - Verify all 4 predictions in ONE forward pass    │
  - ┤
    - Check each token:                               │
    - [fox ✓, jumps ✓, over ✓, the ✗]                 │
    - Accept first 3, reject last                     │
  - ┘
          ↓
Accepted: 3 tokens
Continue with rejected token branch

---

How Speculative Decoding Works

Algorithm: Step by Step

Algorithm: Speculative Decoding

Input:
  prompt: Initial tokens
  target_model: Large LLM (slow, accurate)
  draft_model: Small LLM (fast, approximate)
  K: Number of tokens to speculate (typically 4-8)

Process:
while not finished:
    # Step 1: Draft phase (quick)
    draft_tokens = []
    draft_logits_list = []

    for i in range(K):
        # Use draft model to predict next token
        draft_logits = draft_model(current_sequence)
        draft_tokens.append(argmax(draft_logits))
        draft_logits_list.append(draft_logits)
        current_sequence.append(draft_tokens[-1])

    # Step 2: Verification phase (parallel)
    # Extend sequence with K draft tokens
    extended_sequence = current_sequence + draft_tokens

    # Get target model logits for ALL positions
    target_logits = target_model(extended_sequence)

    # Step 3: Acceptance/Rejection
    accepted_tokens = []
    for i in range(K):
        # Compare draft token with target model's choice
        target_token = argmax(target_logits[-(K-i)])
        draft_token = draft_tokens[i]

        if draft_token == target_token:
            # Accept this token
            accepted_tokens.append(draft_token)
        else:
            # Reject! Use target model's prediction instead
            accepted_tokens.append(target_token)
            break  # Stop here, continue from next iteration

    # Step 4: Output accepted tokens
    output_tokens.extend(accepted_tokens)

Example: Concrete Walkthrough

Scenario:
Prompt: "The quick brown fox"
Target: Llama 2 7B (accurate but slow)
Draft: Llama 2 1B (fast but sometimes wrong)
K = 4 (speculate 4 tokens ahead)

Input sequence: [The, quick, brown, fox] (IDs: [1, 2, 3, 4])

═══════════════════════════════════════════════════════════

Round 1: Speculate & Verify

DRAFT PHASE:
- ┌─ Draft Model processes: [The, quick, brown, fox]
  - Token 5 prediction: "jumps" (ID: 5)
  - Token 6 prediction: "over" (ID: 6)
  - Token 7 prediction: "the" (ID: 7)
  - Token 8 prediction: "lazy" (ID: 8)

Predicted sequence: [The, quick, brown, fox, jumps, over, the, lazy]

VERIFICATION PHASE:
- ┌─ Target Model processes: [The, quick, brown, fox, jumps, over, the, lazy]
  - (all 8 positions at once, in parallel!)
│
  - Position 5: Target predicts "jumps" vs Draft "jumps" → ✓ MATCH!
  - Position 6: Target predicts "over" vs Draft "over" → ✓ MATCH!
  - Position 7: Target predicts "the" vs Draft "the" → ✓ MATCH!
  - Position 8: Target predicts "lazy" vs Draft "lazy" → ✓ MATCH!

RESULT: Accept all 4 tokens!
Output: [jumps, over, the, lazy]
Sequence now: [The, quick, brown, fox, jumps, over, the, lazy]
Cost: 1 draft pass + 1 target pass = 2 passes for 4 tokens
Normal: 4 passes for 4 tokens
Speedup: 2x!

═══════════════════════════════════════════════════════════

Round 2: Speculate & Verify

DRAFT PHASE:
- ┌─ Draft Model processes: [The, quick, brown, fox, jumps, over, the, lazy]
  - Token 9 prediction: "dog" (ID: 9)
  - Token 10 prediction: "." (ID: 10)
  - Token 11 prediction: "End" (ID: End)
  - Token 12 prediction: (not generated, K=4)

Predicted sequence: [The, quick, brown, fox, jumps, over, the, lazy, dog, ., End]

VERIFICATION PHASE:
- ┌─ Target Model processes: [The, quick, brown, fox, jumps, over, the, lazy, dog, ., End]
│
  - Position 9: Target predicts "dog" vs Draft "dog" → ✓ MATCH!
  - Position 10: Target predicts "." vs Draft "." → ✓ MATCH!
  - Position 11: Target predicts "End" vs Draft "End" → ✓ MATCH!
  - Position 12: Not needed (K=3 in last round)

RESULT: Accept all 3 tokens!
Output: [dog, ., End]
Sequence complete!

═══════════════════════════════════════════════════════════

TOTAL COST:
Normal decoding (no speculation):
  - Token 5: 1 pass (50ms)
  - Token 6: 1 pass (50ms)
  - Token 7: 1 pass (50ms)
  - Token 8: 1 pass (50ms)
  - Token 9: 1 pass (50ms)
  - Token 10: 1 pass (50ms)
  - Token 11: 1 pass (50ms)
  - Total: 7 passes = 350ms

Speculative decoding:
  - Round 1: 1 draft pass (10ms) + 1 target pass (50ms) = 60ms, 4 tokens
  - Round 2: 1 draft pass (10ms) + 1 target pass (50ms) = 60ms, 3 tokens
  - Total: 120ms (2.9x faster!)

SPEEDUP: 350ms / 120ms = 2.9x ✓

Why This Works: Probability Distribution

Key insight: Why speculation works!

Each token position has a probability distribution:
P(token | context)

Example at position 5:
P(jumps | [The, quick, brown, fox]) = 0.65  ← Highest prob
P(leaps | [The, quick, brown, fox]) = 0.15
P(runs | [The, quick, brown, fox]) = 0.10
P(other) = 0.10

Both target and draft models estimate this distribution:
  - Target model (7B): Accurate estimate (0.65 for jumps)
  - Draft model (1B): Approximate estimate (0.60 for jumps)

BUT:
  - Both agree on top choice: "jumps"
  - If they agree, we can safely accept
  - If they disagree, target model is more reliable

Empirical observation:
For well-aligned draft/target models:
  - Same top-1 prediction: 80-95% of positions
  - Only verify, don't regenerate
  - Huge speedup with no quality loss!

Why alignment is high:
  - Both models trained on similar data
  - Language structure is predictable
  - Common patterns (e.g., "the" after article)
  - Draft model captures main patterns well

---

Implementing Speculative Decoding

Complete Implementation

import torch
from typing import List, Tuple

class SpeculativeDecoder:
    def __init__(self, 
                 target_model,
                 draft_model,
                 tokenizer,
                 K: int = 4):
        """
        Args:
            target_model: Large accurate model (e.g., Llama 7B)
            draft_model: Small fast model (e.g., Llama 1B)
            tokenizer: Tokenizer for both models
            K: Number of tokens to speculate (4-8 typical)
        """
        self.target = target_model
        self.draft = draft_model
        self.tokenizer = tokenizer
        self.K = K

    def generate(self, 
                 prompt: str, 
                 max_tokens: int = 256) -> str:
        """Generate with speculative decoding"""

        # Tokenize prompt
        input_ids = self.tokenizer(prompt, return_tensors="pt")["input_ids"]
        generated = input_ids.clone()

        tokens_generated = 0
        accepted_tokens = 0
        draft_tokens_count = 0

        while tokens_generated < max_tokens:
            # Step 1: Draft K tokens
            draft_predictions = []
            sequence = generated.clone()

            for i in range(self.K):
                with torch.no_grad():
                    # Draft model predicts next token
                    logits = self.draft(sequence)
                    next_token = logits[:, -1, :].argmax(dim=-1)

                draft_predictions.append(next_token)
                draft_tokens_count += 1
                sequence = torch.cat([sequence, next_token.unsqueeze(-1)], dim=1)

            # Step 2: Verify all K tokens with target model
            with torch.no_grad():
                # Get target model's predictions for extended sequence
                target_logits = self.target(sequence)

            # Step 3: Accept/Reject
            accepted_in_round = 0
            for i in range(self.K):
                target_next = target_logits[:, -(self.K - i), :].argmax(dim=-1)
                draft_next = draft_predictions[i]

                if target_next == draft_next:
                    # Accept draft token
                    generated = torch.cat([
                        generated,
                        draft_next.unsqueeze(-1)
                    ], dim=1)
                    accepted_in_round += 1
                    accepted_tokens += 1
                else:
                    # Use target token instead
                    generated = torch.cat([
                        generated,
                        target_next.unsqueeze(-1)
                    ], dim=1)
                    break  # Stop speculation for this round

            tokens_generated += accepted_in_round + (1 if accepted_in_round < self.K else 0)

        # Decode and return
        output = self.tokenizer.decode(generated[0], skip_special_tokens=True)

        # Print stats
        acceptance_rate = accepted_tokens / draft_tokens_count
        print(f"Acceptance rate: {acceptance_rate:.1%}")

        return output

# Usage
from transformers import AutoModelForCausalLM, AutoTokenizer

target = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
draft = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-1b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

decoder = SpeculativeDecoder(target, draft, tokenizer, K=4)
result = decoder.generate("The quick brown fox", max_tokens=256)
print(result)

Integration with vLLM

from vllm import LLM, SamplingParams

# vLLM with speculative decoding (automatic)
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    # Speculative decoding configuration
    use_v2_block_manager=True,  # Required for spec decode
    speculative_model="meta-llama/Llama-2-1b-hf",  # Draft model
    num_speculative_tokens=4,  # K=4
)

# Generate with speculative decoding enabled
sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=256
)

prompts = ["The quick brown fox"] * 100
outputs = llm.generate(prompts, sampling_params)

# Result: 2-4x faster than without speculation!

---

Performance Analysis

Speedup Calculation

Mathematical analysis:

Let:
  - t_draft = Time for draft model forward pass
  - t_target = Time for target model forward pass
  - K = Number of speculated tokens
  - acceptance_rate = Fraction of draft tokens accepted

Standard decoding for N tokens:
  - Time = N × t_target

Speculative decoding for N tokens:
  - Number of rounds: N / K (roughly)
  - Each round: 1 draft pass + 1 target pass
  - If acceptance_rate = r (0 < r ≤ 1):
    - Effective tokens per round: r × K (on average)
    - Rounds needed: N / (r × K)
  - Time = (N / (r × K)) × (t_draft + t_target)

Speedup:
Speedup = (N × t_target) / ((N / (r × K)) × (t_draft + t_target))
        = (r × K × t_target) / (t_draft + t_target)

Typical values:
  - t_draft ≈ 0.1 × t_target (1B vs 7B model)
  - K = 4
  - r = 0.85 (85% acceptance rate)

Speedup = (0.85 × 4 × t_target) / (0.1 × t_target + t_target)
        = (3.4 × t_target) / (1.1 × t_target)
        = 3.09x

Real-world: 2-4x speedup typical

Real Benchmarks

Model: Llama 2 7B (target) + 1B (draft)
Hardware: A100 GPU
Batch size: 1

                    Standard    Speculative  Speedup
─────────────────────────────────────────────────────
100 tokens          2.5s        0.8s         3.1x
256 tokens          6.4s        2.0s         3.2x
512 tokens          12.8s       4.0s         3.2x
1024 tokens         25.6s       8.0s         3.2x

Acceptance rates:
  - First token: 90%
  - Middle tokens: 85%
  - Last tokens: 80%
  - Average: 85%

Why decreases toward end:
  - Uncertainty increases
  - Draft model becomes less reliable
  - But still maintains 80%+ acceptance

Batch size 32 (vLLM continuous batching):
  - Standard: 3.2 req/sec (256 tokens)
  - Speculative: 10.2 req/sec
  - Speedup: 3.2x maintained even in batch!

---

Draft Model Selection

Choosing the Draft Model

Requirement 1: Speed
  - Must be significantly smaller than target
  - Typical: Target/Draft ratio = 5-7x
  - Example: 7B target + 1B draft = 7x ratio ✓

Requirement 2: Accuracy/Agreement
  - Should agree with target on top-1 prediction
  - Typical agreement: 80-90%
  - If too small: <70% agreement → minimal speedup
  - If too large: Not worth the effort → minimal speedup

Requirement 3: Same Tokenizer
  - Must use same vocabulary
  - Same token IDs for same text
  - Different tokenizers break the algorithm!

Options:

Option A: Smaller Version (BEST)
  - Use smaller version of same model family
  - Example: Llama 2 1B as draft for Llama 2 7B
  - Pros: Same architecture, vocabulary, training data
  - Cons: Limited to available smaller versions
  - Recommendation: Use this when available

Option B: Distilled Model
  - Use knowledge-distilled smaller model
  - Example: DistilBERT for BERT
  - Pros: Highly optimized for speed
  - Cons: Needs special distillation training
  - Recommendation: Good if distilled version exists

Option C: Quantized Draft
  - Use quantized version of same model
  - Example: 7B INT4 as draft for 7B FP16
  - Pros: Same model, different precision
  - Cons: Only 2-3x speedup (not 5-7x)
  - Recommendation: Works if smaller model unavailable

Empirical recommendations:

For Llama family:
  - 70B target: Use 7B or 13B draft (10x or 5x ratio)
  - 13B target: Use 1B or 2B draft (7-13x ratio)
  - 7B target: Use 1B draft (7x ratio)
  - 3B target: INT4 quantize for draft (4x ratio)

For Mistral family:
  - 7B target: Use 1B draft (7x ratio) if available
  - Or use quantized version (4x ratio)

For custom/small models:
  - If no smaller version: Use quantized draft
  - Expect 2-3x speedup instead of 3-4x

---

Practical Considerations

Latency vs Throughput

Speculative Decoding Impact:

LATENCY (single request):
Before: 12.8s (256 tokens)
After: 4.0s (3.2x faster!)
User perceives: Much more responsive ✓

THROUGHPUT (batch):
Before: 32 × 12.8s / 256 = 1.6 req/sec
After: 32 × 4.0s / 256 = 5.1 req/sec (3.2x faster!)
Server capacity: ~3x higher ✓

Ideal use cases:
✓ Single-user interactive (latency matters most)
✓ Batch processing (throughput matters most)
✓ Streaming (both matter)

Not ideal:
✗ When draft model doesn't agree with target
✗ When there's no good draft model available
✗ Very short sequences (overhead dominates)

When Speculation Fails

Low Acceptance Rates (< 70%):
  - Draft model too weak
  - Large domain gap between draft and target
  - Different training/fine-tuning
  - Solution: Retrain draft or use different one

Slow Draft Model:
  - Draft model almost as large as target
  - Not enough speedup to offset overhead
  - Example: 7B draft for 7B target = no benefit
  - Solution: Use smaller draft or quantize more

Tokenizer Mismatch:
  - Different vocabulary or encoding
  - Tokens don't align between models
  - Breaks the entire algorithm!
  - Solution: Use same tokenizer for both

Memory Overhead:
  - Keep both models in memory
  - 7B + 1B = 8GB vs 7B = 3.5GB with quant
  - May exceed GPU memory
  - Solution: Use smaller draft or quantize more

---

Advanced Techniques

Adaptive Speculation

Instead of fixed K, adapt based on context

Observation:
Early tokens in sequence:
  - Easier to predict
  - Higher acceptance rate (90%+)
  - Use K=8 (speculate more)

Middle tokens:
  - Medium difficulty
  - Medium acceptance (85%)
  - Use K=4 (default)

Late tokens:
  - Harder to predict
  - Lower acceptance (70-75%)
  - Use K=2 (speculate less)

Implementation:
```python
def adaptive_K(position: int, sequence_length: int) -> int:
    progress = position / sequence_length

    if progress < 0.25:
        return 8  # Early: speculate aggressively
    elif progress < 0.75:
        return 4  # Middle: default
    else:
        return 2  # Late: conservative

Benefits: - Maintain 3-4x speedup throughout - Avoid wasted speculation when prediction hard - Better resource utilization

### Rejection Sampling with Speculation

Trick: Use speculation for better sampling

Standard sampling: - Generate token from distribution - Some tokens have low probability - Quality depends on random sampling

With speculation: - Generate multiple candidates speculatively - Use target model to score them - Accept highest-quality candidate - Better quality + faster inference!

Implementation:

def speculative_sampling(prompt, num_candidates=4):
    # Generate K candidates with draft
    candidates = []
    for _ in range(num_candidates):
        tokens = draft_model.generate(prompt, max_tokens=1)
        candidates.append(tokens)

    # Score with target model
    scores = []
    for candidate in candidates:
        score = target_model_score(prompt, candidate)
        scores.append(score)

    # Return best
    best_idx = argmax(scores)
    return candidates[best_idx]

Result: - Better output quality (filtering) - Faster than normal sampling (parallel verification) - 2x speedup + better quality!

---

## Comparison with Other Optimizations

Optimization Speedup Compatibility Use When ───────────────────────────────────────────────────────── KV Cache 10x Always ✓ Always Flash Attention 2.8x Most ✓ Long sequences Continuous Batch 2-4x Always ✓ High throughput Quantization 2-3x Always ✓ Memory limited Speculative Decode 2-4x Needs draft ⚠ Has good draft model

### Combined: Speculation + Other Optimizations

Scenario: Llama 2 7B inference

Optimization Stack: - INT4 Quantization: 3.5GB model (2-3x speedup) - KV Cache: Avoid recomputation (10x speedup in practice) - Flash Attention: Fast attention (2.8x speedup) - Continuous Batching: Full GPU (2x throughput) - Speculative Decoding: 1B draft (3x speedup)

Combined effect (multiplicative where independent): - Quantization × KV Cache: Can't multiply (both memory) - KV Cache × Flash Attention: 10x × 1 (attention already efficient) - Batching × Speculation: 2x × 3x = 6x ✓ - All together: Roughly 6x from batching+speculation - + memory efficiency from quantization - Total: 10-20x realistic improvement

Example: Generate 256 tokens, batch size 32

Standard: - 32 requests × 256 tokens = 8192 tokens - Time: 2560 sec (at ~3 tokens/sec, typical) - Cost: $75/hour GPU × 45 min = $56.25

With full stack: - Same 8192 tokens - Time: 256 sec (at ~30 tokens/sec with optimizations) - Cost: $75/hour GPU × 4 min = $5 - Savings: 90% cost reduction + 10x faster ✓

---

## Real-World Deployments

### Deployment Case Study: Long Document QA

Use Case: Answer questions about long documents - Document length: 8K tokens - Question: 100 tokens - Answer target: 256 tokens - Concurrency: 10 users - Latency SLA: < 10 seconds

System Design:

Without Speculation: - Model: Llama 2 7B FP16 (14GB) - Per request: 14GB + 8GB activations = 22GB - Max concurrent: 1-2 requests - Time per request: (8K + 256) / 3 tokens/sec ≈ 45 seconds ❌

With Speculation: - Model: Llama 2 7B INT4 (3.5GB) - Draft: Llama 2 1B INT4 (1GB) - Per request: 4.5GB + 2GB activations = 6.5GB - Max concurrent: 5-6 requests (continuous batching) - Time per request: (8K + 256) / 10 tokens/sec ≈ 15 seconds ✓

Performance: - Latency: 45s → 15s (3x improvement!) ✓ - Throughput: 1 req/sec → 6 req/sec ✓ - Memory: 22GB → 6.5GB (3.4x reduction!) ✓ - Cost: 2× A100s → 1× A100 (50% cost reduction!) ✓

Acceptance Rate Analysis: - First 8K tokens (document): Cached (prefill) - Next 256 tokens (generation): Speculated - Acceptance rate: 85% average - Tokens speculated: 256 × 85% = 217 accepted from 256 drafted = Less than 1 rejection round needed

Confidence: Can meet 10-second SLA with 5-10 concurrent users! ```

---

Best Practices

✅ Do's

1. Pair with good draft model (same family ideally)
2. Monitor acceptance rate (should be 80%+)
3. Combine with other optimizations (quantization, caching)
4. Test on representative workload (different token patterns matter)
5. Keep both models in memory (or fast model-swap)
6. Use for long sequences (speculation value higher)
7. Profile the implementation (verify actual speedup)
8. Handle rejection gracefully (continue from target output)

❌ Don'ts

1. ❌ Use with bad draft model (low agreement)
2. ❌ Assume fixed K works for all contexts
3. ❌ Ignore memory overhead of 2 models
4. ❌ Expect 10x speedup (2-4x realistic)
5. ❌ Use different tokenizers
6. ❌ Apply to very short sequences (< 32 tokens)
7. ❌ Forget to profile on target hardware
8. ❌ Implement naively (be careful with tensor dimensions)

---

Key Takeaways

🔑 Speculative Decoding: 2-4x speedup using draft model prediction
⚡ Parallelizes verification of multiple tokens
💡 Breaks sequential token generation bottleneck
✓ No output quality loss (identical to target model)
🎯 Works best with good draft model (80%+ agreement)
📈 Combines multiplicatively with continuous batching

---

Comparison Summary

Aspect	Standard	With Speculation
Speedup	1x	2-4x
Memory	14GB	14GB + 1GB (draft)
Latency	12.8s (256 tokens)	4.0s
Quality	Baseline	Identical
Complexity	Simple	Medium
Requires	Nothing	Draft model

---

Further Reading

• Speculative Decoding Paper: "Accelerating LLM Inference with Staged Speculative Decoding"
• SGLang Implementation: github.com/hao-ai-lab/sgLang
• vLLM Spec Decode: vllm.readthedocs.io
• Related: Medusa (similar multi-token prediction approach)

---

Related Notes

• Kv Cache - Cache management
• Flash Attention - Fast attention
• Continuous Batching - Dynamic scheduling
• Vllm - Production framework with spec decode
• Llm Inference Optimization - Complete optimization stack