Multimodal & Vision (2017-2023)

Vision models and vision-language models for multimodal AI.

Efficient Vision Architectures

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications - Depthwise separable convolutions
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices - Channel shuffle for efficiency
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks - Principled model scaling

Object Detection & Segmentation

• Mask R-CNN - Instance-level segmentation
• Focal Loss for Dense Object Detection (RetinaNet) - Addressing class imbalance

Vision Transformers

• An Image is Worth 16x16 Words: Transformers for Image Recognition - Vision Transformers (ViT)

Vision-Language Models

• CLIP: Learning Transferable Models for Multimodal Learning - Vision-language alignment
• LLaVA: Large Language and Vision Assistant - Multimodal LLM
• GPT-4V: GPT-4 with Vision Capabilities - Multimodal capabilities
• Flamingo: a Visual Language Model for Few-Shot Learning - In-context multimodal learning

---

Key Insight: Multimodal models enable AI to reason across text and images, opening new applications and capabilities.