Training Language Models to Follow Instructions with Human Feedback

Authors: Ouyang, L., Wu, J., Jiang, X., et al. Year: 2022 Venue: NeurIPS ArXiv: https://arxiv.org/abs/2203.02155

Summary

RLHF (Reinforcement Learning from Human Feedback) uses human preferences as a reward signal to align LLMs with human values. It trained InstructGPT and shaped the foundation of modern helpful, harmless, and honest AI systems.

Key Concepts

• Human preference rankings over model outputs
• Reward model trained on preference data
• Reinforcement learning optimization against reward model
• PPO (Proximal Policy Optimization) algorithm
• Three-stage training pipeline (SFT → Reward Model → RLHF)
• Captures nuanced human preferences
• Enables instruction-following behavior

Impact

RLHF became the standard approach for LLM alignment and safety. It demonstrated that human feedback could steer model behavior at scale, directly enabling ChatGPT's success and the modern era of aligned LLMs.

Related Papers

• Learning to Summarize from Human Feedback
• Proximal Policy Optimization Algorithms
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

---

Citation:

@inproceedings{ouyang2022training,
  title={Training Language Models to Follow Instructions with Human Feedback},
  author={Ouyang, Long and Wu, Jeff and Jiang, Xu and others},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
}