Adapter: Parameter-Efficient Transfer Learning for NLP

Authors: Houlsby, N., Giurgiu, A., Jastrzębski, S., et al. Year: 2019 Venue: EMNLP ArXiv: https://arxiv.org/abs/1902.00751

Summary

Adapters introduce small trainable modules inserted into transformer layers for parameter-efficient transfer learning. They reduce trainable parameters to 0.5-2% of full fine-tuning while maintaining comparable performance.

Key Concepts

• Small bottleneck layers in transformer blocks
• Parameter reduction without full retraining
• Faster training and inference than full fine-tuning
• Multiple adapters composable for task combinations
• Adapter weights shareable across models
• Foundation for later LoRA and prefix tuning

Impact

Adapters pioneered the parameter-efficient fine-tuning paradigm. While LoRA later proved more practical for LLMs, adapters established key principles for efficient transfer learning still used today.

Related Papers

• LoRA: Low-Rank Adaptation
• Prefix Tuning: Optimizing Continuous Prompts

---

Citation:

@inproceedings{houlsby2019parameter,
  title={Parameter-Efficient Transfer Learning for NLP},
  author={Houlsby, Neil and Giurgiu, Andrei and Jastrzębski, Stanisław and others},
  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  year={2019}
}