LoRA: Low-Rank Adaptation of Large Language Models

Authors: Hu et al. Year: 2021 ArXiv/Link: https://arxiv.org/abs/2106.09685

Summary

Low-rank decomposition enables efficient fine-tuning of large models by adding small trainable matrices to frozen weights.

Key Concepts

• Low-rank adaptation
• Parameter efficiency
• Fine-tuning
• Rank decomposition
• Additive adapters

Impact

Enabled practical fine-tuning of billion-parameter models

Category

Fine-tuning & Adaptation

---