Distilling the Knowledge in a Neural Network¶
Authors: Hinton et al. Year: 2015 ArXiv/Link: https://arxiv.org/abs/1503.02531
Summary¶
Transfer knowledge from large models to smaller models through soft targets and temperature scaling.
Key Concepts¶
- Knowledge distillation
- Model compression
- Soft targets
- Temperature scaling
- Student-teacher learning
Impact¶
Foundational technique for model compression
Category¶
Knowledge Distillation