What Is a Mixture-of-Experts Model? A Plain-English Explainer
Some AI models are built from many smaller “experts” and a router that picks a few of them for each word. What that means, why it makes a big model cheaper to run, and the catch that matters if you want to run one yourself.
SkillsUSA California State Champion, Information Technology Services (ITS)
3 min read
On this page
Read about AI models for long enough and you’ll run into “MoE”, or a name like “8x7B”, or a line saying a model has one number of “total” parameters and a much smaller number of “active” ones. Those all point to the same idea: a Mixture-of-Experts model.
You don’t need to know how to build one. But the idea explains why some large models are surprisingly quick to run, and why they can still be hard to fit on your own hardware. Here it is in plain English.
The usual way: every part works on every word
Most AI models are “dense.” The Switch Transformer paper puts it simply: “In deep learning, models typically reuse the same parameters for all inputs.” Every word you send goes through every part of the model. A bigger model can know more, but it also does more work for every single word, which makes it slower and more expensive to run.
The Mixture-of-Experts way: a router picks a few
A Mixture-of-Experts (MoE) model splits some of its layers into several parallel blocks called experts, and adds a small router in front of them. Text is processed in small pieces called tokens (roughly a word or part of a word). For each token, the router picks the few experts that will handle it, and only those experts do any work for that token. The rest sit idle for it.
- The
- invoice
- is
- overdue
- Expert 1
- Expert 2
- Expert 3
- Expert 4
- Expert 5
- Expert 6
- Expert 7
- Expert 8
The picks shown are illustrative. In a real model the router is learned in training, and it chooses again at every Mixture-of-Experts layer.
A well-known open example is Mixtral 8x7B. Its paper describes eight experts in each layer, and for every token, at each layer, “a router network selects two experts to process the current state and combine their outputs.” The result: “each token has access to 47B parameters, but only uses 13B active parameters during inference.” Google’s earlier Switch Transformer used an even simpler rule: each token goes to just one expert.
The idea itself isn’t new. A 2017 paper from Google researchers built on an older one, “conditional computation, where parts of the network are active on a per-example basis,” and used it to make models far larger without a matching increase in the work done for each input.
“Experts” aren’t specialists in the way it sounds
The name suggests one expert for invoices, another for legal questions and another for code. That isn’t how it works. Nobody assigns the experts their jobs; the router learns which experts to use while the model is trained, and it chooses again at every MoE layer.
When the Mixtral team looked at which experts handled which text, they wrote: “Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.” The choices lined up more with the structure of the text than with its subject. So don’t read too much into the word “expert”. It’s a part of the model, not a department.
Have a task your team repeats every week? Let’s map it and see whether automating it pays.
Plan my automationWhy it matters for a business
It can make a large model cheaper to run per request. Because only a fraction of the model works on each token, an MoE model can answer faster than a dense model with the same total number of parameters. The Hugging Face explainer lists “faster inference compared to a model with the same number of parameters” as a main benefit.
But it still needs memory for the whole thing. Every expert has to be loaded, even the ones a given token doesn’t use. In Hugging Face’s words, MoE models “require high VRAM as all experts are loaded in memory.” If you want to run one on your own hardware, plan around the total size, not the active size. A model with 47 billion parameters needs memory for 47 billion parameters, even if it only uses about 13 billion for each token. Shrinking the weights with quantization helps; What Is Quantization? explains how.
It’s a design choice, not a quality label. “MoE” on a model card tells you how the model is built, not whether it’s right for your work. Test any model on real examples from your own business before you rely on it.
If you’re comparing models for a project, or deciding between a hosted service and something you run yourself, I can help you sort out what actually matters for your use. That’s part of the business automation work I do.
Sources
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Shazeer et al., 2017
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, Fedus, Zoph and Shazeer, 2021
- Mixtral of Experts, Jiang et al., 2024: eight experts per layer, two per token, total and active parameters, and the routing analysis
- Mixture of Experts Explained, Hugging Face