RoBERTa: A Robustly Optimized BERT Pretraining Approach¶
Authors: Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., et al. Year: 2019 ArXiv: https://arxiv.org/abs/1907.11692
Summary¶
RoBERTa improves BERT by modifying key hyperparameters and training procedures. It removes NSP objective, trains on larger data for longer, and uses dynamic masking patterns, achieving state-of-the-art on GLUE and SuperGLUE benchmarks.
Key Concepts¶
- Improved training procedures over BERT
- Removal of NSP objective
- Dynamic masking patterns
- Larger batch sizes and longer training
- Better downstream task performance
Impact¶
RoBERTa showed that pre-training details matter as much as architecture. It set the standard for how to properly train transformer encoders and became the baseline for many subsequent improvements.
Related Papers¶
- BERT: Pre-training of Deep Bidirectional Transformers
- ELECTRA: Pre-training Text Encoders as Discriminators
Citation:
@inproceedings{liu2019roberta,
title={RoBERTa: A Robustly Optimized BERT Pretraining Approach},
author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and others},
booktitle={arXiv preprint arXiv:1907.11692},
year={2019}
}