Supervised Fine-Tuning of Large Language Models

Authors: Ouyang, L., Wu, J., Jiang, X., et al. Year: 2022 Venue: NeurIPS ArXiv: https://arxiv.org/abs/2203.02155

Summary

Supervised Fine-Tuning (SFT) is the foundation of instruction-following LLMs. It involves training models on high-quality labeled examples of task completions, enabling models to follow diverse instructions without requiring reinforcement learning.

Key Concepts

• High-quality instruction-completion pairs
• Task diversity and coverage
• Supervised learning on desired outputs
• Foundation for RLHF and reward modeling
• Data quality critical for performance
• Enables zero-shot generalization to new tasks
• Cost-effective compared to reinforcement learning

Impact

SFT established the paradigm for creating instruction-following models (InstructGPT, ChatGPT). It's the essential first step before RLHF, proving that task supervision alone drives dramatic capability improvements.

Related Papers

• Training Language Models to Follow Instructions with Human Feedback
• Scaling Instruction-Finetuned Language Models
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

---

Citation:

@inproceedings{ouyang2022training,
  title={Training Language Models to Follow Instructions with Human Feedback},
  author={Ouyang, Long and Wu, Jeff and Jiang, Xu and others},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
}