Learning to Summarize from Human Feedback

Authors: Zellers, R., Holtzman, A., Peters, M. E., et al. Year: 2020 Venue: NeurIPS ArXiv: https://arxiv.org/abs/2009.06032

Summary

This paper introduces the foundational approach for using human feedback to improve LLM outputs, specifically for summarization. It demonstrates that reward models can capture human preferences and guide policy optimization.

Key Concepts

• Reward models trained from human preferences
• Policy gradient optimization
• Preference learning from pairwise comparisons
• Foundation for modern RLHF pipeline
• Human feedback improves factuality and relevance
• Establishes feasibility of feedback-driven learning at scale

Impact

This work laid the groundwork for RLHF, proving human feedback could effectively steer model behavior. It directly informed the development of InstructGPT and modern alignment approaches.

Related Papers

• Training Language Models to Follow Instructions with Human Feedback
• Direct Preference Optimization

---

Citation:

@inproceedings{zellers2020learning,
  title={Learning to Summarize from Human Feedback},
  author={Zellers, Rowan and Holtzman, Ari and Peters, Matthew E and others},
  booktitle={Advances in Neural Information Processing Systems},
  year={2020}
}