Skip to content

Latent Dirichlet Allocation (Topic Modeling)

Authors: David M. Blei, Andrew Y. Ng, Michael I. Jordan Year: 2003 Venue: JMLR 2003 Citations: 30,000+

Summary

Introduces Latent Dirichlet Allocation (LDA), a generative probabilistic model for collections of discrete data. Models each document as mixture of topics, each topic as distribution over words. Foundation for unsupervised topic discovery in text.

Key Concepts

  • Topic Modeling: Discovers latent topics in document collections
  • Latent Variables: Topics and document-topic assignments
  • Dirichlet Priors: Symmetric priors on topic and word distributions
  • Bayesian Inference: Uses variational inference or Gibbs sampling
  • Exchangeability: Conditional independence structure
  • Interpretability: Learned topics interpretable as word distributions

Impact

  • 30,000+ citations
  • Foundation of topic modeling and unsupervised text analysis
  • Widely used for document analysis and information retrieval
  • Inspired Bayesian non-parametric models (hierarchical Dirichlet processes)
  • Connected Bayesian methods with NLP
  • Standard baseline for text analysis
  • Applications in document recommendation, exploration

Key Results

  • Discovered interpretable topics in large document collections
  • Better generalization than previous topic models
  • Practical inference algorithms (variational, Gibbs sampling)
  • Scalable to large collections
  • Latent Semantic Analysis (Deerwester et al., 1990)
  • Probabilistic Latent Semantic Indexing (Hofmann, 1999)
  • Hierarchical Dirichlet Process (Teh et al., 2006)
  • Word2Vec (Mikolov et al., 2013)