Language Models are Unsupervised Multitask Learners

Authors: Radford et al. Year: 2019 ArXiv/Link: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Summary

GPT-2 demonstrated that large-scale unsupervised language modeling learns multiple NLP tasks without explicit supervision, introducing few-shot learning capabilities.

Key Concepts

• Unsupervised multitask learning
• In-context learning
• Autoregressive decoding
• Few-shot performance
• Large-scale pretraining

Impact

Demonstrated emergent multitask learning in language models

Category

Transformers

---