Adam: A Method for Stochastic Optimization

Authors: Kingma, D. P., & Ba, J. Year: 2014 ArXiv: https://arxiv.org/abs/1412.6980

Summary

Adam combines the benefits of AdaGrad and RMSProp by maintaining both first and second moment estimates of gradients. It's one of the most widely used optimizers in deep learning due to its adaptive learning rates and efficiency.

Key Concepts

• Adaptive learning rate per parameter
• First moment (mean) and second moment (variance) estimates
• Bias correction for moment estimates
• Combines momentum and RMSProp advantages
• Robust to hyperparameter choices

Impact

Adam has become the default optimizer for most deep learning applications, replacing SGD as the standard choice. Its adaptive nature makes it suitable for sparse gradients and non-stationary problems.

Related Papers

• RMSProp: Divide the gradient by a running average of its recent magnitude
• AdaGrad: Adaptive Subgradient Methods for Online Learning

---

Citation:

@inproceedings{kingma2014adam,
  title={Adam: A method for stochastic optimization},
  author={Kingma, Diederik P and Ba, Jimmy},
  booktitle={arXiv preprint arXiv:1412.6980},
  year={2014}
}