Deep contextualized word representations (ELMo)

Authors: Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, et al. Year: 2018 Venue: NAACL 2018 Citations: 15,000+

Link

• ELMo: Contextualized Embeddings (Peters et al., 2018)

Summary

Introduces ELMo (Embeddings from Language Models), deep contextualized word representations from bidirectional LSTMs. Word representations depend on context, not fixed per word. Improves over static embeddings for NLP tasks before BERT.

Key Concepts

• Contextualized Embeddings: Word representation depends on context
• Bidirectional LSTMs: Captures both forward and backward context
• Character-Based CNNs: Works with character inputs
• Transfer Learning: Pre-trained representations transfer to downstream tasks
• Attention: Task-specific attention over LSTM layers
• Deep Representation: Uses all layers of language model

Impact

• 15,000+ citations
• Major advance over static word embeddings
• Inspired BERT, GPT, and modern contextualized embeddings
• Demonstrated power of pre-trained language models
• Enabled transfer learning for NLP
• Practical improvements across NLP benchmarks
• Foundation for modern NLP before Transformers

Key Results

• Significant improvements on diverse NLP tasks
• Better than Word2Vec and GloVe
• Pre-trained models transfer well
• Interpretable attention patterns

Related Papers

• Word2Vec and Static Embeddings (Mikolov et al., 2013)
• BERT: Bidirectional Transformers (Devlin et al., 2018)
• GPT: Generative Pre-trained Transformer (Radford et al., 2018)
• Language Models (Bengio et al., 2003)