Mixtral of Experts

Authors: Jiang et al. Year: 2024 ArXiv/Link: https://arxiv.org/abs/2401.04088

Summary

Mixture of Experts language model combining 8 expert networks with gating mechanisms for improved efficiency and capabilities.

Key Concepts

• Mixture of experts
• Sparse routing
• Gating mechanisms
• Expert networks
• Conditional computation

Impact

Enabled efficient conditional computation for LLMs

Category

Open Source Models

---