What is a mixture of experts model, and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. Mixture of experts from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits inside the transformer architecture learning path and builds on the basics of neural networks.

Why interviewers ask this

The question separates candidates who know the slogan from those who know the costs. Reported by candidates in loops at AI labs and large tech companies, the prompts range from "explain how a mixture of experts works" to routing, the balancing loss and serving a large model with a latency target. The topic matters because recent language models such as Mixtral and DeepSeek-V3 use the design.

A weak answer says "the model has many experts and uses some of them." A strong answer says what an expert is, how a router picks, why a balancing term is needed and why memory does not shrink. A Staff-level answer adds the serving consequence: the experts must be spread across several graphics processing units (GPUs), so tokens travel between devices, and the trade-off becomes speed against communication.

Mixture of experts from first principles

Mixture of experts

A mixture of experts is a model layout in which one large layer is replaced by several smaller expert networks and a router that chooses, for each input, which experts process it. The outputs of the chosen experts are combined into one result, so capacity grows without every part running on every input.

Router

A router is a small learned layer that scores every expert for a given token and selects the best few. Its scores also set how much each chosen expert counts in the final output.

In plain words

Picture a hospital with a front desk and eight specialist doctors. Every patient meets the desk first, and the desk sends each one to two doctors, whose opinions are combined. The hospital employs eight doctors but each patient uses only two. The analogy breaks in one place: all eight doctors must stay on site and paid, because the next patient may need any of them. That is the memory point of this page. Also, the experts are not human-style subjects; the authors of the Mixtral paper report no obvious topic-based pattern in which tokens go to which expert.

In one precise sentence: a mixture of experts layer computes a weighted sum of a few expert outputs, where a learned router picks the experts and sets the weights.

The idea is old. Jacobs et al. (1991) trained separate networks that each handle part of the data, with a gating network choosing among them, and Jordan and Jacobs (1994) stacked the gates into a hierarchy.

Step 1: The router scores every expert for one token

Inside a transformer layer, attention mixes information between tokens first. Then, in an ordinary dense model, one feed-forward network processes every token. A dense model is one in which every weight is used for every token.

In a mixture of experts layer, that single feed-forward network is replaced by several copies with different weights, the experts. The router, a single matrix multiplication, takes the token's vector and produces one score per expert. Input: one token vector. Output: a list of expert scores. Attention layers stay shared and unchanged; see self-attention and multi-head attention for that part. You now know where the experts sit: only in the feed-forward position of each layer.

Step 2: Top-k routing picks a few experts and mixes their outputs

Top-k routing keeps the k highest router scores and ignores the rest. Shazeer et al. (2017) introduced this sparsely-gated layer, with up to thousands of feed-forward experts and a mixture of experts with up to 137 billion parameters. Fedus et al. (2021) simplified it to a single expert per token in the Switch Transformer, and Mixtral uses two of eight experts in every layer.

In Mixtral, the weights come from a softmax over the kept scores, so they sum to one; Switch instead takes a softmax over all experts and keeps the top probability. Input: expert scores. Output: the chosen experts and their weights. The layer output is the weighted sum of the chosen experts' outputs. With illustrative scores of 2.0, 1.0, 0.5 and -1.0 for four experts, the top two get weights 0.7311 and 0.2689.¹

Step 3: Only the chosen experts compute, but all experts stay in memory

Only the chosen experts run for a token, so compute per token follows the active parameters, not the total. That is the whole saving, and it is real: a model can hold far more knowledge capacity than it pays to run.

The error is to read this as a memory saving. The next token in the batch may choose different experts, so the server cannot know in advance which weights it can leave out. Every expert must be loaded and ready. In the worked example below, an illustrative layout with two of eight experts has 46.70 billion total parameters but 12.88 billion active ones, which is 27.6% of the total, yet loading it in 16-bit numbers still takes 93.41 GB.¹ Compute shrinks to about a quarter; memory does not shrink at all.

Step 4: A load-balancing loss keeps the experts evenly used

A load-balancing loss is an extra training term that penalizes the router for sending most tokens to a few experts. Without it, training tends to favor early winners: a favored expert gets more updates, gets better and gets chosen more, until the rest are wasted.

Fedus et al. (2021) define it as the number of experts times the sum, over experts, of the fraction of tokens sent to the expert times the average router probability for it. It equals 1 when use is perfectly even and grows as use concentrates. Input: the router's choices over a batch. Output: one number added to the training loss. A different fix is expert-choice routing, in which Zhou et al. (2022) let each expert pick its tokens, so every expert gets the same load. In practice, systems also cap how many tokens each expert accepts per batch.

1 · Router scores2 · Top-2 mix3 · All in memory4 · Balance lossReplay

Step-by-step description

1. The router scores all eight experts.
2. The top two experts compute and are mixed.
3. Six idle experts still stay loaded.
4. A loss keeps expert use even.

• The router scores every expert for one token.
• Only the chosen experts compute and their outputs are mixed by softmax weights.
• All experts stay in memory because the next token may choose any of them.
• A load-balancing loss keeps the router from favoring a few experts.

Worked example

Take one layer layout, labeled illustrative, that is the configuration reported for Mixtral 8x7B in Table 1 of the Mixtral paper: 32 layers, a model width of 4,096, an expert hidden width of 14,336, eight experts per layer with each expert a gated block of three weight matrices, two experts chosen per token, a vocabulary of 32,000 and attention with 8 shared key-value heads of width 128. The sizes follow by arithmetic.¹

One expert has 176,160,768 weights. Eight experts in one layer hold 1,409,286,144, and 32 layers hold 45.10 billion. Attention adds 1.34 billion, the input and output embeddings add 0.26 billion and the routers add about 1 million, so the total is 46.70 billion parameters, ignoring the small normalization weights. Per token, only two experts per layer run: 11.27 billion from experts plus the same shared parts gives 12.88 billion active.¹ These match the totals of 47 billion and 13 billion that the Mixtral abstract reports.

The last row uses an illustrative batch where four experts receive 50%, 25%, 12.5% and 12.5% of tokens with average router probabilities 0.4, 0.3, 0.2 and 0.1; the value is 1.25, so the penalty rises as use concentrates.¹

Text description of the diagram

• The illustrative layout has 46.70 billion total parameters.
• A token uses 12.88 billion active parameters, which is 27.6% of the total.

Trade-offs and edge cases

A mixture of experts trades memory and engineering complexity for compute. Against a dense model of the same total size, it computes less per token; against a dense model with the same active parameters, it stores more and often reaches better quality, which is why the design is popular.

Text description of the diagram

• A dense model uses every weight for every token.
• A mixture of experts uses a few experts per token but loads all of them.
• Serving a mixture of experts adds communication between devices.

Edge cases that interviewers probe:

• Serving: experts are usually spread over several GPUs, a layout called expert parallelism, so tokens move between devices after routing. In practice communication can outweigh the saved compute at small batch sizes; that is a rule of thumb, not a measured figure, and the Switch Transformers abstract names communication costs among the known difficulties.
• Batching: a single request may touch only a few experts, but a large batch touches nearly all, so the savings depend on traffic.
• Training stability: routers can collapse onto a few experts, which the balancing loss and token caps fight.
• Hard routing: the choice of experts is not differentiable. Gradients reach the router through the softmax weights of the chosen experts, not through the choice itself.
• Newer layouts: DeepSeekMoE splits experts into many small ones and keeps a few shared experts always on (Dai et al., 2024); DeepSeek-V3 has 671 billion total parameters with 37 billion active per token (DeepSeek-AI, 2024).

Attention cost is separate from all of this; see FlashAttention for that side of efficiency.

How to answer it in an interview

The spoken structure: say what is replaced, say how the router picks, give the compute saving, then name memory, communication and balancing.

Common mistakes

• "A mixture of experts needs less memory because it runs fewer experts." (wrong) The next token may need any expert. (right) "It needs less compute per token, but all experts must be loaded."
• "Each expert is a specialist in a topic like law or code." (wrong) The Mixtral authors report no clear topic pattern. (right) "Experts specialize in ways that are not neatly human-readable."
• "A mixture of experts is an ensemble of separate models." (wrong) The experts sit inside each layer and share attention. (right) "It is one model with a sparse feed-forward block."
• "The router just picks the best expert, so there is no gradient problem." (wrong) The choice is a hard selection. (right) "Gradients flow through the weights of the chosen experts."

Follow-up questions

What is the mixture of experts in LLM?

A mixture of experts in a large language model (LLM) replaces the feed-forward block of each transformer layer with several experts and a router, so the model has many parameters but few active per token. The LLM interview questions page covers neighboring topics.

Mixture of experts vs dense model?

A dense model uses every weight for every token, which is simple to train and serve. A mixture of experts uses a few experts per token, which lowers compute per token but keeps the memory bill and adds routing and communication.

Who introduced the mixture of experts?

Jacobs et al. (1991) introduced adaptive mixtures of local experts, and Shazeer et al. (2017) brought the sparse version into deep language models.

Why use two of eight experts?

Choosing one expert is cheapest, as in the Switch Transformer; choosing two lets the layer mix two views and gives the router a smoother signal. The exact reason a given lab chose its setting is a design judgment, so present it as a trade-off, not a rule.

How do you make a hard router learnable?

Keep the top-k choice but multiply each chosen expert's output by its softmax weight, so the loss gradient reaches the router through those weights. Shazeer et al. add tunable noise to the scores, which helps balance load, and expert-choice routing is a different way to even out load.

How would you serve a large mixture of experts model?

Spread experts across GPUs, batch requests so most experts get work, watch the token traffic between devices and set latency targets per percentile. Memory sizing follows total parameters, not active ones.

Sources

1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv 1701.06538) — arXiv (Shazeer et al.), 23 Jan 2017
2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (arXiv 2101.03961) — arXiv (Fedus, Zoph, Shazeer), 11 Jan 2021
3. Mixtral of Experts (arXiv 2401.04088) — arXiv (Jiang et al., Mistral AI), 8 Jan 2024
4. Adaptive Mixtures of Local Experts — Neural Computation (Jacobs, Jordan, Nowlan, Hinton), 1 Mar 1991
5. Hierarchical Mixtures of Experts and the EM Algorithm — Neural Computation (Jordan and Jacobs), 1 Mar 1994
6. Mixture-of-Experts with Expert Choice Routing (arXiv 2202.09368) — arXiv (Zhou et al.), 18 Feb 2022
7. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (arXiv 2401.06066) — arXiv (Dai et al.), 11 Jan 2024
8. DeepSeek-V3 Technical Report (arXiv 2412.19437) — arXiv (DeepSeek-AI), 27 Dec 2024