RMSProp: Neural Networks for Machine Learning

Authors: Tieleman, T., & Hinton, G. Year: 2012 Course Notes: Coursera lecture notes

Summary

RMSProp (Root Mean Square Propagation) is an unpublished, adaptive learning rate method that divides the learning rate by an exponentially decaying average of squared gradients. Developed independently and popularized through Coursera lectures.

Key Concepts

• Exponentially decaying average of squared gradients
• Adaptive per-parameter learning rates
• Handles non-stationary objectives well
• Prevents learning rate from decaying to zero
• Efficient for mini-batch training

Impact

RMSProp became a popular optimizer for RNNs and sparse data, especially in deep reinforcement learning. It inspired many subsequent adaptive methods like Adam.

Related Papers

• Adam: A Method for Stochastic Optimization
• AdaGrad: Adaptive Subgradient Methods

---

Citation:

@misc{tieleman2012rmsprop,
  title={Lecture 6e: RMSProp},
  author={Tieleman, Tieleman and Hinton, Geoffrey},
  year={2012},
  howpublished={Coursera Lecture Notes}
}