Skip to content

On the Importance of Initialization and Momentum in Deep Learning

Authors: Ilya Sutskever, James Martens, George Dahl, Geoffrey E. Hinton Year: 2013 Venue: ICML 2013 Citations: 10,000+

Summary

Studies the importance of initialization and momentum for training deep networks. Shows momentum (especially Nesterov) significantly accelerates training. Provides practical guidance for initialization schemes and optimizer choices.

Key Concepts

  • Initialization Schemes: Careful weight initialization matters for convergence
  • Momentum: Accumulates gradient updates enabling faster convergence
  • Nesterov Momentum: Look-ahead gradient improves convergence rate
  • Initialization Sensitivity: Deep networks sensitive to starting weights
  • Convergence Speed: Momentum substantially accelerates training
  • Practical Guidelines: Empirical recommendations for practitioners

Impact

  • 10,000+ citations
  • Foundational understanding of optimization for deep networks
  • Motivated momentum-based optimizers (Adam, RMSProp, etc.)
  • Practical guidance for training deep networks
  • Influenced initialization schemes (Xavier, He)
  • Important for understanding convergence
  • Essential knowledge for practitioners

Key Results

  • Momentum significantly accelerates convergence
  • Nesterov momentum outperforms standard momentum
  • Initialization scheme impacts training speed
  • Practical recommendations for hyperparameters
  • Adam Optimizer (Kingma & Ba, 2015)
  • Xavier Initialization (Glorot & Bengio, 2010)
  • He Initialization (He et al., 2015)
  • Understanding Optimization (Nocedal & Wright, 2006)