Deep Learning Era (2012-2016)

The revolution that transformed AI through deep neural networks and large-scale learning.

Image Classification

• ImageNet Large Scale Visual Recognition Challenge - Dataset driving the revolution
• ImageNet Classification with Deep CNNs (AlexNet) - Deep CNN winning ImageNet 2012
• Very Deep Convolutional Networks for Large-Scale Image Recognition (VGGNet) - Systematic depth study
• Deep Residual Learning for Image Recognition (ResNet) - Skip connections for very deep networks
• Densely Connected Convolutional Networks (DenseNet) - Dense layer connections
• Inception-v4, Inception-ResNet - Combining Inception with residuals

Object Detection

• Rich feature hierarchies for object detection (R-CNN) - Region-based detection
• Faster R-CNN: Towards Real-Time Object Detection - Region proposal networks
• You Only Look Once (YOLO): Unified Real-Time Object Detection - Single-stage detection
• SSD: Single Shot MultiBox Detector - Multi-scale detection

Training Fundamentals

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting - Stochastic regularization
• Batch Normalization: Accelerating Deep Network Training - Layer normalization
• On the Importance of Initialization and Momentum in Deep Learning - Optimization insights
• Adam: A Method for Stochastic Optimization - Adaptive learning rates

Word Embeddings & NLP

• Efficient Estimation of Word Representations in Vector Space (Word2Vec) - Skip-gram embeddings
• GloVe: Global Vectors for Word Representation - Global statistics

Generative Models

• Generative Adversarial Networks (GANs) - Two competing networks
• Auto-Encoding Variational Bayes (Variational Autoencoders) - Probabilistic generative model

Reinforcement Learning

• Playing Atari with Deep Reinforcement Learning (DQN) - Deep Q-Networks
• Asynchronous Methods for Deep Reinforcement Learning (A3C) - Parallel RL
• Mastering the Game of Go with Deep Neural Networks (AlphaGo) - Deep learning + MCTS

Meta-Learning & Few-Shot

• Model-Agnostic Meta-Learning for Fast Adaptation (MAML) - Learning to learn
• Prototypical Networks for Few-Shot Learning - Metric learning

Graph & Attention Methods

• Semi-Supervised Classification with Graph Convolutional Networks (GCN) - Convolutions on graphs
• Inductive Representation Learning on Large Graphs (GraphSAGE) - Inductive GNN
• Squeeze-and-Excitation Networks - Channel attention

Contrastive & Federated Learning

• A Simple Framework for Contrastive Learning (SimCLR) - Self-supervised learning
• Momentum Contrast for Unsupervised Visual Representation (MoCo) - Contrastive learning
• Communication-Efficient Learning of Deep Networks (Federated Averaging) - Privacy-preserving learning

Contextual Representations

• Deep Contextualized Word Representations (ELMo) - Bidirectional LSTM embeddings

Classical ML at Scale

• XGBoost: A Scalable Tree Boosting System - Production-grade gradient boosting

---

Key Insight: This era established that deep learning at scale could solve real-world problems better than classical methods.