Skip to content

Efficient Estimation of Word Representations in Vector Space (Word2Vec)

Authors: Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean Year: 2013 Venue: arXiv:1301.3781 & ICCV 2013 Workshop Citations: 60,000+

Summary

Introduces Word2Vec, two efficient neural network architectures for learning high-quality word embeddings. Skip-gram predicts context from target word, CBOW predicts target from context. Uses negative sampling for efficient training on large corpora.

Key Concepts

  • Word Embeddings: Dense vector representations capturing semantic/syntactic relationships
  • Skip-gram Model: Predicts surrounding words from target word
  • CBOW Model: Predicts target word from context words
  • Negative Sampling: Efficient training avoiding expensive softmax
  • Hierarchical Softmax: Tree-based softmax for computation efficiency

Impact

  • 60,000+ citations - revolutionized NLP
  • Simple models capturing deep linguistic patterns
  • Enabled transfer learning via embeddings
  • Word vectors encode semantic relationships (king - man + woman ≈ queen)
  • Basis for modern NLP systems

Key Results

  • Fast training (millions of words/second on CPU)
  • Better semantic accuracy than previous methods
  • Demonstrates power of efficient training for representation learning
  • GloVe: Global Vectors (Pennington et al., 2014)
  • FastText: Subword Information (Bojanowski et al., 2017)
  • Neural Language Models (Bengio et al., 2003)