BERT: Pre-training of Deep Bidirectional Transformers

Authors: Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. Year: 2018 ArXiv: https://arxiv.org/abs/1810.04805

Summary

BERT introduces bidirectional pre-training of transformers using masked language modeling (MLM) and next sentence prediction (NSP). It achieves state-of-the-art results on 11 NLP tasks with minimal fine-tuning.

Key Concepts

• Bidirectional context modeling (unlike GPT)
• Masked language modeling (MLM) objective
• Next sentence prediction (NSP) task
• Efficient fine-tuning on downstream tasks
• Encoder-only architecture

Impact

BERT revolutionized NLP by showing that bidirectional pre-training outperforms left-to-right models. It inspired hundreds of domain-specific BERT variants and became the foundation for transfer learning in NLP.

Related Papers

• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• ELECTRA: Pre-training Text Encoders as Discriminators
• Attention Is All You Need

---

Citation:

@inproceedings{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle={arXiv preprint arXiv:1810.04805},
  year={2018}
}