Prefix Tuning: Optimizing Continuous Prompts for Generation

Authors: Li, X. L., & Liang, P. Year: 2021 Venue: ACL ArXiv: https://arxiv.org/abs/2101.00190

Summary

Prefix Tuning learns task-specific continuous vectors prepended to the model input, achieving parameter efficiency through learned prompts rather than fine-tuning weights. It demonstrates that prompt learning can match full fine-tuning.

Key Concepts

• Continuous learnable prompt vectors
• Freezes model parameters
• Task-specific prefix prepended to input
• 0.1% of model parameters trainable
• Comparable performance to full fine-tuning
• Foundation for prompt-based adaptation
• Applicable to both encoder and decoder models

Impact

Prefix Tuning pioneered the parameter-efficient prompt-learning approach. While later surpassed by LoRA for large models, it established that prompt learning could be competitive, influencing modern prompt engineering practices.

Related Papers

• LoRA: Low-Rank Adaptation
• Adapter: Parameter-Efficient Transfer Learning

---

Citation:

@inproceedings{li2021prefix,
  title={Prefix Tuning: Optimizing Continuous Prompts for Generation},
  author={Li, Xiang Lisa and Liang, Percy},
  booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics},
  year={2021}
}