Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Authors: Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. Year: 2023 Venue: NeurIPS ArXiv: https://arxiv.org/abs/2305.18290

Summary

DPO eliminates the need for explicit reward models by directly optimizing the language model using preference pairs. It achieves comparable or better results than RLHF while being simpler, more stable, and more computationally efficient.

Key Concepts

• Direct optimization from preference data
• No explicit reward model needed
• Closed-form solution to alignment objective
• Simpler pipeline than RLHF (SFT → DPO instead of SFT → RM → RLHF)
• Improved stability and convergence
• Lower computational requirements
• Bradley-Terry model for preferences

Impact

DPO simplified LLM alignment by proving reward models weren't strictly necessary. It became widely adopted as a faster, simpler alternative to RLHF, enabling more frequent model iterations and democratizing alignment techniques.

Related Papers

• Training Language Models to Follow Instructions with Human Feedback
• Learning to Summarize from Human Feedback
• Offline RL for Language Models: Learning to Rank for Question Answering

---

Citation:

@inproceedings{rafailov2023direct,
  title={Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and others},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023}
}