Skip to content

A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)

Authors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton Year: 2020 Venue: ICML 2020 Citations: 15,000+

Summary

Introduces SimCLR, a simple yet effective contrastive learning framework for self-supervised learning. Uses data augmentation, projections, and contrastive loss to learn visual representations. Without labels, achieves competitive performance with supervised methods.

Key Concepts

  • Self-Supervised Learning: Learn representations without labels
  • Contrastive Loss: Maximize similarity between augmented views of same image
  • Data Augmentation: Critical component of contrastive methods
  • Projection Head: Non-linear projection for contrastive loss
  • Momentum Contrast: Maintains queue of negative samples
  • Pre-training: Unsupervised pre-training enabling efficient fine-tuning

Impact

  • 15,000+ citations
  • Sparked self-supervised learning revolution in vision
  • Inspired MoCo, BYOL, SwAV, and many variants
  • Practical alternative to supervised pre-training
  • Reduced labeled data requirements for training
  • Influenced NLP self-supervised methods (ELECTRA, etc.)
  • Critical technique in modern vision pipelines

Key Results

  • Self-supervised learning competitive with ImageNet supervised pre-training
  • Scaling benefits with larger models
  • Strong transfer learning performance
  • Better generalization than supervised models
  • Momentum Contrast (MoCo, He et al., 2020)
  • Bootstrap Your Own Latent (BYOL, Grill et al., 2020)
  • Contrastive Divergence (Hinton, 2002)
  • Metric Learning (Siamese networks, Koch et al., 2015)