Speculative Decoding for Faster Large Language Model Inference¶
Authors: Chen et al. Year: 2023 ArXiv/Link: https://arxiv.org/abs/2302.01318
Summary¶
Uses a smaller model to predict multiple tokens speculatively, verified by larger model for speedups without quality loss.
Key Concepts¶
- Speculative decoding
- Multi-token prediction
- Smaller verifier
- Speedup
- Token efficiency
Impact¶
Practical speedup technique for autoregressive generation
Category¶
Decoding Strategies