BERT: Pre-training of Deep Bidirectional Transformers¶
Authors: Devlin et al. Year: 2018 ArXiv/Link: https://arxiv.org/abs/1810.04805
Summary¶
Introduced masked language modeling (MLM) and bidirectional pre-training for transformers, enabling effective transfer learning for NLP tasks.
Key Concepts¶
- Masked language modeling
- Bidirectional encoding
- Pre-training and fine-tuning
- NSP (Next Sentence Prediction)
- Transfer learning
Impact¶
Pioneered pre-trained language models for downstream NLP tasks
Category¶
Transformers