RoFormer: Enhanced Transformer with Rotary Position Embedding¶
Authors: Su et al. Year: 2021 ArXiv/Link: https://arxiv.org/abs/2104.09864
Summary¶
Introduced Rotary Position Embeddings (RoPE), encoding absolute position using rotation matrix with excellent extrapolation properties.
Key Concepts¶
- Rotary embeddings
- Position encoding
- Extrapolation
- Relative position bias
- Complex embeddings
Impact¶
Improved position encoding with better long-context generalization
Category¶
Position Embeddings