Speculative Decoding

Overview

Speculative Decoding is an inference optimization technique that uses a smaller draft model to generate multiple predicted tokens ahead, then verifies them in parallel with a large target model. This breaks the sequential bottleneck of token generation, achieving 2-4x speedup without changing model outputs.

• Paper: "Faster Transformer Decoding through Non-Autoregressive Iterative Refinement"
• Key Innovation: Parallel verification of predicted tokens
• Impact: 2-4x speedup, especially for long-context and batch inference
• Complexity: Medium (algorithm is straightforward but careful implementation needed)
• Adoption: Growing in vLLM, SGLang, and other inference engines

-

The Problem: Sequential Token Generation Bottleneck

Why Token Generation is Slow

Standard Autoregressive Decoding (Sequential):

For generating N tokens:
 - Token 1: Forward pass → get output
 - Token 2: Forward pass → get output
 - Token 3: Forward pass → get output
 -...
 - Token N: Forward pass → get output

Key observation:
 - Each token requires ONE forward pass
 - Passes cannot be parallelized (each depends on previous)
 - Must wait for each token to start next
 - LATENCY BOTTLENECK! (can't parallelize)

Example: Generate 256 tokens
Llama 2 7B inference:
 - Model size: 14GB (FP16)
 - Forward pass: 50ms (on A100)
 - Total time: 256 × 50ms = 12.8 seconds
 - But GPU is idle between passes!
 - Reason: Memory-bound operation + sequential dependency

Why it's memory-bound:
 - Load 14GB model weights
 - Process 1 token (tiny computation)
 - Memory bandwidth limits speed
 - Can't parallelize due to dependencies

Conclusion:
Each token takes ~50ms (not because of compute,
but because of memory bandwidth + sequential structure)

The Sequential Dependency Problem

Standard Autoregressive Generation:

Input: "The quick brown"

Step 1:
 - Input: [The, quick, brown]
 - Forward pass: Process all 3 tokens
 - Output: Prediction for token 4 = "fox"
 - Time: 50ms

Step 2:
 - Input: [The, quick, brown, fox]
 - Forward pass: Process all 4 tokens ← REDUNDANT!
 - Output: Prediction for token 5 = "jumps"
 - Time: 50ms

Step 3:
 - Input: [The, quick, brown, fox, jumps]
 - Forward pass: Process all 5 tokens ← MORE REDUNDANT!
 - Output: Prediction for token 6 = "over"
 - Time: 50ms

Observations:
1. Recompute tokens 1-3 again and again (with KV cache, stored)
2. Each step must wait for previous to complete
3. Only the LAST token's output matters
4. First N-1 tokens' predictions are ignored!
5. This is inherently sequential (can't parallelize)

With KV cache optimization:
 - Don't recompute, just reuse (good)
 - But still must generate ONE token per pass
 - Still sequential (bad)
 - Still 256 passes for 256 tokens

Performance Ceiling

Memory Bandwidth Analysis:

A100 GPU Memory Bandwidth: 2 TB/s
Llama 2 7B Model: 14GB weights (FP16)

Per token:
 - Load model: 14GB
 - Process: 1 token (minimal computation)
 - Effective tokens/sec: 2 TB/s / (14 × 10^9 bytes) = 142 tokens/sec
 - BUT with overhead: ~100-120 tokens/sec in practice
 - Time per token: 1/100 = 10ms

But we measure 50ms per token (5x worse!)
Why?
 - Not 100% memory bandwidth utilization
 - Other overheads (scheduling, I/O)
 - Kernel launch overhead
 - Result: ~50ms per token

For 256 tokens: 256 × 50ms = 12.8 seconds

Speculative Decoding changes this!

-

Speculative Decoding: The Solution

Core Idea: Predict Ahead, Verify in Parallel

Key Insight:
"Instead of generating 1 token per pass,
 generate K tokens speculatively,
 then verify all K in parallel!"

Standard:
Pass 1 → Token 1 → Pass 2 → Token 2 → Pass 3 → Token 3...
Total: N passes for N tokens (sequential)

Speculative:
Draft: Predict tokens 1-4 quickly
Verify: Check all 4 with large model in parallel 
Result: ~N/4 passes for N tokens (4x speedup!)

Architecture:
- ┌──────────────────────────────────┐
 - Input Prompt │
 - ┘
 ↓
- ┌──────────────────────────────────┐
 - Draft Model (small, fast) │
 - Generate K=4 predicted tokens │
 - ┤
 - Predictions: │
 - [fox, jumps, over, the] │
 - ┘
 ↓
- ┌──────────────────────────────────────────────────┐
 - Target Model (large, slow) │
 - Verify all 4 predictions in ONE forward pass │
 - ┤
 - Check each token: │
 - [fox , jumps , over , the ] │
 - Accept first 3, reject last │
 - ┘
 ↓
Accepted: 3 tokens
Continue with rejected token branch

-

How Speculative Decoding Works

Algorithm: Step by Step

Algorithm: Speculative Decoding

Input:
 prompt: Initial tokens
 target_model: Large LLM (slow, accurate)
 draft_model: Small LLM (fast, approximate)
 K: Number of tokens to speculate (typically 4-8)

Process:
while not finished:
 # Step 1: Draft phase (quick)
 draft_tokens = []
 draft_logits_list = []

 for i in range(K):
 # Use draft model to predict next token
 draft_logits = draft_model(current_sequence)
 draft_tokens.append(argmax(draft_logits))
 draft_logits_list.append(draft_logits)
 current_sequence.append(draft_tokens[-1])

 # Step 2: Verification phase (parallel)
 # Extend sequence with K draft tokens
 extended_sequence = current_sequence + draft_tokens

 # Get target model logits for ALL positions
 target_logits = target_model(extended_sequence)

 # Step 3: Acceptance/Rejection
 accepted_tokens = []
 for i in range(K):
 # Compare draft token with target model's choice
 target_token = argmax(target_logits[-(K-i)])
 draft_token = draft_tokens[i]

 if draft_token == target_token:
 # Accept this token
 accepted_tokens.append(draft_token)
 else:
 # Reject! Use target model's prediction instead
 accepted_tokens.append(target_token)
 break # Stop here, continue from next iteration

 # Step 4: Output accepted tokens
 output_tokens.extend(accepted_tokens)

Example: Concrete Walkthrough

Scenario:
Prompt: "The quick brown fox"
Target: Llama 2 7B (accurate but slow)
Draft: Llama 2 1B (fast but sometimes wrong)
K = 4 (speculate 4 tokens ahead)

Input sequence: [The, quick, brown, fox] (IDs: [1, 2, 3, 4])

═══════════════════════════════════════════════════════════

Round 1: Speculate & Verify

DRAFT PHASE:
- ┌─ Draft Model processes: [The, quick, brown, fox]
 - Token 5 prediction: "jumps" (ID: 5)
 - Token 6 prediction: "over" (ID: 6)
 - Token 7 prediction: "the" (ID: 7)
 - Token 8 prediction: "lazy" (ID: 8)

Predicted sequence: [The, quick, brown, fox, jumps, over, the, lazy]

VERIFICATION PHASE:
- ┌─ Target Model processes: [The, quick, brown, fox, jumps, over, the, lazy]
 - (all 8 positions at once, in parallel!)
│
 - Position 5: Target predicts "jumps" vs Draft "jumps" → MATCH!
 - Position 6: Target predicts "over" vs Draft "over" → MATCH!
 - Position 7: Target predicts "the" vs Draft "the" → MATCH!
 - Position 8: Target predicts "lazy" vs Draft "lazy" → MATCH!

RESULT: Accept all 4 tokens!
Output: [jumps, over, the, lazy]
Sequence now: [The, quick, brown, fox, jumps, over, the, lazy]
Cost: 1 draft pass + 1 target pass = 2 passes for 4 tokens
Normal: 4 passes for 4 tokens
Speedup: 2x!

═══════════════════════════════════════════════════════════

Round 2: Speculate & Verify

DRAFT PHASE:
- ┌─ Draft Model processes: [The, quick, brown, fox, jumps, over, the, lazy]
 - Token 9 prediction: "dog" (ID: 9)
 - Token 10 prediction: "." (ID: 10)
 - Token 11 prediction: "End" (ID: End)
 - Token 12 prediction: (not generated, K=4)

Predicted sequence: [The, quick, brown, fox, jumps, over, the, lazy, dog,., End]

VERIFICATION PHASE:
- ┌─ Target Model processes: [The, quick, brown, fox, jumps, over, the, lazy, dog,., End]
│
 - Position 9: Target predicts "dog" vs Draft "dog" → MATCH!
 - Position 10: Target predicts "." vs Draft "." → MATCH!
 - Position 11: Target predicts "End" vs Draft "End" → MATCH!
 - Position 12: Not needed (K=3 in last round)

RESULT: Accept all 3 tokens!
Output: [dog,., End]
Sequence complete!

═══════════════════════════════════════════════════════════

TOTAL COST:
Normal decoding (no speculation):
 - Token 5: 1 pass (50ms)
 - Token 6: 1 pass (50ms)
 - Token 7: 1 pass (50ms)
 - Token 8: 1 pass (50ms)
 - Token 9: 1 pass (50ms)
 - Token 10: 1 pass (50ms)
 - Token 11: 1 pass (50ms)
 - Total: 7 passes = 350ms

Speculative decoding:
 - Round 1: 1 draft pass (10ms) + 1 target pass (50ms) = 60ms, 4 tokens
 - Round 2: 1 draft pass (10ms) + 1 target pass (50ms) = 60ms, 3 tokens
 - Total: 120ms (2.9x faster!)

SPEEDUP: 350ms / 120ms = 2.9x 

Why This Works: Probability Distribution

Key insight: Why speculation works!

Each token position has a probability distribution:
P(token| context)

Example at position 5:
P(jumps| [The, quick, brown, fox]) = 0.65 ← Highest prob
P(leaps| [The, quick, brown, fox]) = 0.15
P(runs| [The, quick, brown, fox]) = 0.10
P(other) = 0.10

Both target and draft models estimate this distribution:
 - Target model (7B): Accurate estimate (0.65 for jumps)
 - Draft model (1B): Approximate estimate (0.60 for jumps)

BUT:
 - Both agree on top choice: "jumps"
 - If they agree, we can safely accept
 - If they disagree, target model is more reliable

Empirical observation:
For well-aligned draft/target models:
 - Same top-1 prediction: 80-95% of positions
 - Only verify, don't regenerate
 - Huge speedup with no quality loss!

Why alignment is high:
 - Both models trained on similar data
 - Language structure is predictable
 - Common patterns (e.g., "the" after article)
 - Draft model captures main patterns well

-

Implementing Speculative Decoding

Complete Implementation

import torch
from typing import List, Tuple

class SpeculativeDecoder:
 def __init__(self, 
 target_model,
 draft_model,
 tokenizer,
 K: int = 4):
 """
 Args:
 target_model: Large accurate model (e.g., Llama 7B)
 draft_model: Small fast model (e.g., Llama 1B)
 tokenizer: Tokenizer for both models
 K: Number of tokens to speculate (4-8 typical)
 """
 self.target = target_model
 self.draft = draft_model
 self.tokenizer = tokenizer
 self.K = K

 def generate(self, 
 prompt: str, 
 max_tokens: int = 256) -> str:
 """Generate with speculative decoding"""

 # Tokenize prompt
 input_ids = self.tokenizer(prompt, return_tensors="pt")["input_ids"]
 generated = input_ids.clone()

 tokens_generated = 0
 accepted_tokens = 0
 draft_tokens_count = 0

 while tokens_generated < max_tokens:
 # Step 1: Draft K tokens
 draft_predictions = []
 sequence = generated.clone()

 for i in range(self.K):
 with torch.no_grad():
 # Draft model predicts next token
 logits = self.draft(sequence)
 next_token = logits[:, -1,:].argmax(dim=-1)

 draft_predictions.append(next_token)
 draft_tokens_count += 1
 sequence = torch.cat([sequence, next_token.unsqueeze(-1)], dim=1)

 # Step 2: Verify all K tokens with target model
 with torch.no_grad():
 # Get target model's predictions for extended sequence
 target_logits = self.target(sequence)

 # Step 3: Accept/Reject
 accepted_in_round = 0
 for i in range(self.K):
 target_next = target_logits[:, -(self.K - i),:].argmax(dim=-1)
 draft_next = draft_predictions[i]

 if target_next == draft_next:
 # Accept draft token
 generated = torch.cat([
 generated,
 draft_next.unsqueeze(-1)
], dim=1)
 accepted_in_round += 1
 accepted_tokens += 1
 else:
 # Use target token instead
 generated = torch.cat([
 generated,
 target_next.unsqueeze(-1)
], dim=1)
 break # Stop speculation for this round

 tokens_generated += accepted_in_round + (1 if accepted_in_round < self.K else 0)

 # Decode and return
 output = self.tokenizer.decode(generated[0], skip_special_tokens=True)

 # Print stats
 acceptance_rate = accepted_tokens / draft_tokens_count
 print(f"Acceptance rate: {acceptance_rate:.1%}")

 return output

# Usage
from transformers import AutoModelForCausalLM, AutoTokenizer

target = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
draft = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-1b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

decoder = SpeculativeDecoder(target, draft, tokenizer, K=4)
result = decoder.generate("The quick brown fox", max_tokens=256)
print(result)

Integration with vLLM

from vllm import LLM, SamplingParams

# vLLM with speculative decoding (automatic)
llm = LLM(
 model="meta-llama/Llama-2-7b-hf",
 # Speculative decoding configuration
 use_v2_block_manager=True, # Required for spec decode
 speculative_model="meta-llama/Llama-2-1b-hf", # Draft model
 num_speculative_tokens=4, # K=4
)

# Generate with speculative decoding enabled
sampling_params = SamplingParams(
 temperature=0.7,
 max_tokens=256
)

prompts = ["The quick brown fox"] * 100
outputs = llm.generate(prompts, sampling_params)

# Result

-

Performance Analysis

Speedup Calculation

Mathematical analysis:

Let:
 - t_draft = Time for draft model forward pass
 - t_target = Time for target model forward pass
 - K = Number of speculated tokens
 - acceptance_rate = Fraction of draft tokens accepted

Standard decoding for N tokens:
 - Time = N × t_target

Speculative decoding for N tokens:
 - Number of rounds: N / K (roughly)
 - Each round: 1 draft pass + 1 target pass
 - If acceptance_rate = r (0 < r ≤ 1):
 - Effective tokens per round: r × K (on average)
 - Rounds needed: N / (r × K)
 - Time = (N / (r × K)) × (t_draft + t_target)

Speedup:
Speedup = (N × t_target) / ((N / (r × K)) × (t_draft + t_target))
 = (r × K × t_target) / (t_draft + t_target)

Typical values:
 - t_draft ≈ 0.1 × t_target (1B vs 7B model)
 - K = 4
 - r = 0.85 (85% acceptance rate)

Speedup = (0.85 × 4 × t_target) / (0.1 × t_target + t_target)
 = (3.4 × t_target) / (1.1 × t_target)
 = 3.09x

Real-world: 2-4x speedup typical

Real Benchmarks

Model: Llama 2 7B (target) + 1B (draft)
Hardware: A100 GPU
Batch size: 1

 Standard Speculative Speedup
─────────────────────────────────────────────────────
100 tokens 2.5s 0.8s 3.1x
256 tokens 6.4s 2.0s 3.2x
512 tokens 12.8s 4.0s 3.2x
1024 tokens 25.6s 8.0s 3.2x

Acceptance rates:
 - First token: 90%
 - Middle tokens: 85%
 - Last tokens: 80%
 - Average: 85%

Why decreases toward end:
 - Uncertainty increases
 - Draft model becomes less reliable
 - But still maintains 80%+ acceptance

Batch size 32 (vLLM continuous batching):
 - Standard: 3.2 req/sec (256 tokens)
 - Speculative: 10.2 req/sec
 - Speedup: 3.2x maintained even in batch!

-

Draft Model Selection

Choosing the Draft Model

Requirement 1: Speed
 - Must be significantly smaller than target
 - Typical: Target/Draft ratio = 5-7x
 - Example: 7B target + 1B draft = 7x ratio 

Requirement 2: Accuracy/Agreement
 - Should agree with target on top-1 prediction
 - Typical agreement: 80-90%
 - If too small: <70% agreement → minimal speedup
 - If too large: Not worth the effort → minimal speedup

Requirement 3: Same Tokenizer
 - Must use same vocabulary
 - Same token IDs for same text
 - Different tokenizers break the algorithm!

Options:

Option A: Smaller Version (BEST)
 - Use smaller version of same model family
 - Example: Llama 2 1B as draft for Llama 2 7B
 - Pros: Same architecture, vocabulary, training data
 - Cons: Limited to available smaller versions
 - Recommendation: Use this when available

Option B: Distilled Model
 - Use knowledge-distilled smaller model
 - Example: DistilBERT for BERT
 - Pros: Highly optimized for speed
 - Cons: Needs special distillation training
 - Recommendation: Good if distilled version exists

Option C: Quantized Draft
 - Use quantized version of same model
 - Example: 7B INT4 as draft for 7B FP16
 - Pros: Same model, different precision
 - Cons: Only 2-3x speedup (not 5-7x)
 - Recommendation: Works if smaller model unavailable

Empirical recommendations:

For Llama family:
 - 70B target: Use 7B or 13B draft (10x or 5x ratio)
 - 13B target: Use 1B or 2B draft (7-13x ratio)
 - 7B target: Use 1B draft (7x ratio)
 - 3B target: INT4 quantize for draft (4x ratio)

For Mistral family:
 - 7B target: Use 1B draft (7x ratio) if available
 - Or use quantized version (4x ratio)

For custom/small models:
 - If no smaller version: Use quantized draft
 - Expect 2-3x speedup instead of 3-4x

-

Practical Considerations

Latency vs Throughput

Speculative Decoding Impact:

LATENCY (single request):
Before: 12.8s (256 tokens)
After: 4.0s (3.2x faster!)
User perceives: Much more responsive 

THROUGHPUT (batch):
Before: 32 × 12.8s / 256 = 1.6 req/sec
After: 32 × 4.0s / 256 = 5.1 req/sec (3.2x faster!)
Server capacity: ~3x higher 

Ideal use cases:
Single-user interactive (latency matters most)
Batch processing (throughput matters most)
Streaming (both matter)

Not ideal:
When draft model doesn't agree with target
When there's no good draft model available
Very short sequences (overhead dominates)

When Speculation Fails

Low Acceptance Rates (< 70%):
 - Draft model too weak
 - Large domain gap between draft and target
 - Different training/fine-tuning
 - Solution: Retrain draft or use different one

Slow Draft Model:
 - Draft model almost as large as target
 - Not enough speedup to offset overhead
 - Example: 7B draft for 7B target = no benefit
 - Solution: Use smaller draft or quantize more

Tokenizer Mismatch:
 - Different vocabulary or encoding
 - Tokens don't align between models
 - Breaks the entire algorithm!
 - Solution: Use same tokenizer for both

Memory Overhead:
 - Keep both models in memory
 - 7B + 1B = 8GB vs 7B = 3.5GB with quant
 - May exceed GPU memory
 - Solution: Use smaller draft or quantize more

-

Advanced Techniques

Adaptive Speculation

Instead of fixed K, adapt based on context

Observation:
Early tokens in sequence:
 - Easier to predict
 - Higher acceptance rate (90%+)
 - Use K=8 (speculate more)

Middle tokens:
 - Medium difficulty
 - Medium acceptance (85%)
 - Use K=4 (default)

Late tokens:
 - Harder to predict
 - Lower acceptance (70-75%)
 - Use K=2 (speculate less)

Implementation:
```python
def adaptive_K(position: int, sequence_length: int) -> int:
 progress = position / sequence_length

 if progress < 0.25:
 return 8 # Early: speculate aggressively
 elif progress < 0.75:
 return 4 # Middle: default
 else:
 return 2 # Late: conservative

Benefits:

• Maintain 3-4x speedup throughout
• Avoid wasted speculation when prediction hard
• Better resource utilization

### Rejection Sampling with Speculation

Trick: Use speculation for better sampling

Standard sampling:

• Generate token from distribution
• Some tokens have low probability
• Quality depends on random sampling

With speculation:

• Generate multiple candidates speculatively
• Use target model to score them
• Accept highest-quality candidate
• Better quality + faster inference!

Implementation:

def speculative_sampling(prompt, num_candidates=4):
 # Generate K candidates with draft
 candidates = []
 for _ in range(num_candidates):
 tokens = draft_model.generate(prompt, max_tokens=1)
 candidates.append(tokens)

 # Score with target model
 scores = []
 for candidate in candidates:
 score = target_model_score(prompt, candidate)
 scores.append(score)

 # Return best
 best_idx = argmax(scores)
 return candidates[best_idx]

Result:

• Better output quality (filtering)
• Faster than normal sampling (parallel verification)
• 2x speedup + better quality!

-

## Comparison with Other Optimizations

Optimization Speedup Compatibility Use When ───────────────────────────────────────────────────────── KV Cache 10x Always Always Flash Attention 2.8x Most Long sequences Continuous Batch 2-4x Always High throughput Quantization 2-3x Always Memory limited Speculative Decode 2-4x Needs draft Has good draft model

### Combined: Speculation + Other Optimizations

Scenario: Llama 2 7B inference

Optimization Stack:

• INT4 Quantization: 3.5GB model (2-3x speedup)
• KV Cache: Avoid recomputation (10x speedup in practice)
• Flash Attention: Fast attention (2.8x speedup)
• Continuous Batching: Full GPU (2x throughput)
• Speculative Decoding: 1B draft (3x speedup)

Combined effect (multiplicative where independent):

• Quantization × KV Cache: Can't multiply (both memory)
• KV Cache × Flash Attention: 10x × 1 (attention already efficient)
• Batching × Speculation: 2x × 3x = 6x
• All together: Roughly 6x from batching+speculation
• • memory efficiency from quantization
• Total: 10-20x realistic improvement

Example: Generate 256 tokens, batch size 32

Standard:

• 32 requests × 256 tokens = 8192 tokens
• Time: 2560 sec (at ~3 tokens/sec, typical)
• Cost: $75/hour GPU × 45 min = $56.25

With full stack:

• Same 8192 tokens
• Time: 256 sec (at ~30 tokens/sec with optimizations)
• Cost: $75/hour GPU × 4 min = $5
• Savings: 90% cost reduction + 10x faster

-

## Real-World Deployments

### Deployment Case Study: Long Document QA

Use Case: Answer questions about long documents

• Document length: 8K tokens
• Question: 100 tokens
• Answer target: 256 tokens
• Concurrency: 10 users
• Latency SLA: < 10 seconds

System Design:

Without Speculation:

• Model: Llama 2 7B FP16 (14GB)
• Per request: 14GB + 8GB activations = 22GB
• Max concurrent: 1-2 requests
• Time per request: (8K + 256) / 3 tokens/sec ≈ 45 seconds

With Speculation:

• Model: Llama 2 7B INT4 (3.5GB)
• Draft: Llama 2 1B INT4 (1GB)
• Per request: 4.5GB + 2GB activations = 6.5GB
• Max concurrent: 5-6 requests (continuous batching)
• Time per request: (8K + 256) / 10 tokens/sec ≈ 15 seconds

Performance:

• Latency: 45s → 15s (3x improvement!)
• Throughput: 1 req/sec → 6 req/sec
• Memory: 22GB → 6.5GB (3.4x reduction!)
• Cost: 2× A100s → 1× A100 (50% cost reduction!)

Acceptance Rate Analysis:

• First 8K tokens (document): Cached (prefill)
• Next 256 tokens (generation): Speculated
• Acceptance rate: 85% average
• Tokens speculated: 256 × 85% = 217 accepted from 256 drafted = Less than 1 rejection round needed

Confidence: Can meet 10-second SLA with 5-10 concurrent users!

```

---

Best Practices

Do's

1. Pair with good draft model (same family ideally)
2. Monitor acceptance rate (should be 80%+)
3. Combine with other optimizations (quantization, caching)
4. Test on representative workload (different token patterns matter)
5. Keep both models in memory (or fast model-swap)
6. Use for long sequences (speculation value higher)
7. Profile the implementation (verify actual speedup)
8. Handle rejection gracefully (continue from target output)

Don'ts

1. Use with bad draft model (low agreement)
2. Assume fixed K works for all contexts
3. Ignore memory overhead of 2 models
4. Expect 10x speedup (2-4x realistic)
5. Use different tokenizers
6. Apply to very short sequences (< 32 tokens)
7. Forget to profile on target hardware
8. Implement naively (be careful with tensor dimensions)

---

Key Takeaways

Speculative Decoding: 2-4x speedup using draft model prediction Parallelizes verification of multiple tokens Breaks sequential token generation bottleneck No output quality loss (identical to target model) Works best with good draft model (80%+ agreement) Combines multiplicatively with continuous batching

---

Comparison Summary

Aspect	Standard	With Speculation
Speedup	1x	2-4x
Memory	14GB	14GB + 1GB (draft)
Latency	12.8s (256 tokens)	4.0s
Quality	Baseline	Identical
Complexity	Simple	Medium
Requires	Nothing	Draft model

-

Further Reading

• Speculative Decoding Paper: "Accelerating LLM Inference with Staged Speculative Decoding"
• SGLang Implementation: github.com/hao-ai-lab/sgLang
• vLLM Spec Decode: vllm.readthedocs.io
• Related: Medusa (similar multi-token prediction approach)

-

Related Notes

• Kv Cache - Cache management
• Flash Attention - Fast attention
• Continuous Batching - Dynamic scheduling
• Vllm - Production framework with spec decode
• Llm Inference Optimization - Complete optimization stack