Fixing Weight Decay Regularization in Adam

Authors: Loshchilov, I., & Hutter, F. Year: 2019 ArXiv: https://arxiv.org/abs/1711.05101

Summary

AdamW decouples weight decay from gradient-based update, fixing a fundamental issue in Adam where L2 regularization becomes ineffective. This simple fix significantly improves generalization in modern deep learning.

Key Concepts

• Weight decay decoupled from gradient updates
• Fixes L2 regularization in adaptive optimizers
• Improves generalization significantly
• Enables proper regularization in Adam
• Widely adopted in transformer training

Impact

AdamW has become the standard optimizer for transformer models (BERT, GPT, etc.), replacing vanilla Adam. The fix is simple but crucial for proper regularization in modern architectures.

Related Papers

• Adam: A Method for Stochastic Optimization
• Fixing Weight Decay Regularization in Adam

---

Citation:

@inproceedings{loshchilov2019decoupled,
  title={Decoupled Weight Decay Regularization},
  author={Loshchilov, Ilya and Hutter, Frank},
  booktitle={International Conference on Learning Representations},
  year={2019}
}