LoRA: Low-Rank Adaptation of Large Language Models

Authors: Hu, E. J., Shen, Y., Wallis, P., et al. Year: 2021 Venue: ICLR ArXiv: https://arxiv.org/abs/2106.09685

Summary

LoRA introduces a parameter-efficient fine-tuning method by freezing pre-trained model weights and adding trainable low-rank matrices to each layer. It reduces trainable parameters by up to 10,000x while maintaining or exceeding full fine-tuning performance.

Key Concepts

• Low-rank decomposition (LoRA matrices A and B)
• Frozen pre-trained weights
• Scalable fine-tuning for large models
• 10,000x parameter reduction
• No inference latency overhead
• Composable adapters for multiple tasks

Impact

LoRA revolutionized LLM fine-tuning, enabling efficient adaptation on consumer hardware. It became the foundation for models like Alpaca, Vicuña, and established the adapter paradigm. Now industry-standard for enterprise LLM customization.

Related Papers

• QLoRA: Efficient Finetuning of Quantized LLMs
• Adapter: Parameter-Efficient Transfer Learning
• Prefix Tuning: Optimizing Continuous Prompts

---

Citation:

@inproceedings{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J and Shen, Yelong and Wallis, Phillip and others},
  booktitle={International Conference on Learning Representations},
  year={2021}
}