Skip to content

The EM Algorithm: Maximum Likelihood from Incomplete Data

Authors: Arthur P. Dempster, Nan M. Laird, Donald B. Rubin Year: 1977 Venue: Journal of the Royal Statistical Society Citations: 50,000+

Summary

Introduces the Expectation-Maximization (EM) algorithm for maximum likelihood estimation with missing or latent variables. Iteratively estimates parameters by computing expected sufficient statistics (E-step) and maximizing likelihood (M-step). Foundation for many statistical learning methods.

Key Concepts

  • Incomplete Data Problem: Missing or unobserved latent variables
  • E-Step: Compute expected log-likelihood given current parameters
  • M-Step: Maximize likelihood w.r.t. parameters
  • Latent Variables: Unknown variables modeling hidden structure
  • Convergence: Monotonically increases likelihood
  • Generality: Applies to diverse models (mixtures, HMMs, factor analysis)

Impact

  • 50,000+ citations
  • Foundational algorithm in statistical machine learning
  • Basis for mixture models, hidden Markov models, latent factor models
  • Used in clustering (K-means is special case of EM)
  • Theoretical foundation for variational inference
  • Still widely used in probabilistic modeling
  • Essential for understanding many ML algorithms

Key Results

  • Systematic algorithm for incomplete data learning
  • Convergence guarantee (monotonic likelihood increase)
  • Practical algorithm for complex models
  • Applicable to many probabilistic models
  • Mixture Models (McLachlan & Basford, 1988)
  • Hidden Markov Models (Rabiner, 1989)
  • Variational Inference (Jordan et al., 1999)
  • Mean-Field Variational Inference (Hoffman et al., 2013)