ALiBi: Train Short, Test Long

Authors: Press et al. Year: 2022 ArXiv/Link: https://arxiv.org/abs/2108.12409

Summary

Attention with Linear Biases (ALiBi) directly adds bias to attention scores, enabling models trained on short sequences to extrapolate to longer ones.

Key Concepts

• Linear biases
• Attention bias
• Length extrapolation
• Minimal position information
• Simple yet effective

Impact

Enabled length generalization without explicit position embeddings

Category

Position Embeddings

---