In this article
Before you start
This page sits in the large language model (LLM) training learning path, and it assumes you have met the transformer architecture once.
Why interviewers ask this
Mixtral 8x7B is the model that made mixture of experts a common topic: after its release, Hugging Face wrote that this class of transformer had become the hottest topic in the open AI community. So a question about it tests whether you understand sparse models, not whether you can recite a name.
The interviewer is looking for the gap between two numbers. A weak answer says "eight experts of seven billion each, so 56B parameters, and it runs like a 13B model, so it is cheap." A strong answer explains why the total is closer to 47B, why only some weights run per token, and why the memory still has to hold all of them. A Staff-level answer adds what follows: serving cost, load balancing in training, and how fine-tuning differs from a dense model.
Mixtral 8x7B from first principles
- Mixture of experts (MoE)
- A mixture-of-experts (MoE) layer is a block made of several separate sub-networks, called experts, plus a router that sends each token to only a few of them. A model can then hold many parameters while running only some of them for each token.
- Active parameters
- Active parameters are the weights that take part in the computation for one token. In a sparse model they are fewer than the total parameters, and all the total parameters still stay in memory.
In plain words
Picture a hospital with eight specialist doctors and one triage nurse. Each patient sees only two doctors. The nurse picks them, and the two opinions are blended. The hospital is cheap per patient, because only two doctors work on each one. But all eight doctors must be on the payroll.
The analogy breaks in one place. The experts are not trained as named specialists. The router and the experts learn together, and the choice is made again for every token at every layer (Jiang et al., 2024).
In one precise sentence, Mixtral 8x7B is a transformer whose feed-forward blocks are each replaced by 8 experts and a router, so each token uses 2 experts per layer.
Step 1: Each layer swaps one feed-forward block for 8 experts
Mixtral 8x7B has the same architecture as Mistral 7B, except that each layer is composed of 8 feedforward blocks, called experts. Only the feed-forward network (FFN) is multiplied. Attention and the other parameters stay shared, because the architecture is otherwise the same as Mistral 7B (Jiang et al., 2024).
That is why the name misleads. Eight copies of a 7B model would be 56B parameters, but only the feed-forward part is copied, so the total is about 47B.1 Input: the token's hidden state after attention. Output: nothing yet, since the experts have not run. You now know the shape: one shared attention block, then 8 parallel expert blocks. The router in the next step decides which of them run.
Step 2: A router picks 2 experts and mixes their outputs
For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. The usual sparse gate scores all experts, keeps the top few and applies a softmax to the kept scores, as in the Shazeer et al. (2017) design the Hugging Face post describes.
Take illustrative router scores for 8 experts: 2.1, 0.3, 1.4, -0.5, 0.0, 0.9, -1.2 and 0.2. The two highest are expert 1 (2.1) and expert 3 (1.4). A softmax over just those two gives weights of 0.6682 and 0.3318.1 The layer output is 0.6682 times expert 1's output plus 0.3318 times expert 3's.
Input: a hidden state and 8 scores. Output: one mixed vector. The other 6 experts do no work for this token. The next token can pick a different pair.
Step 3: Compute follows 13B but memory follows 47B
Each token sees 2 experts per layer, so each token has access to 47B parameters but only uses 13B active parameters during inference. Compute per token therefore resembles a 13B dense model. Hugging Face's explainer estimates the per-token compute at roughly that of a 12B model rather than 2 × 7B = 14B, because the shared layers run once. The 13B is the paper's active-parameter count; the 12B is a rough compute estimate.
Memory does not shrink. Video random-access memory (VRAM) holds the model weights on the graphics processing unit (GPU), and all experts are loaded in memory, so the card must hold a 47B model. The next token may need any expert, so none can be left out.
At 2 bytes per parameter (fp16), 47B parameters take 94 gigabytes (GB), while a 13B model would need 26 GB.1 This is the term candidates tend to forget: sparse means cheap compute, not small weights. You now have the whole mechanism: shared attention, 8 experts, a router picking 2, and a memory cost set by the total.
Step-by-step description
- Attention is shared; the feed-forward block becomes 8 experts.
- The router keeps 2 experts and mixes their outputs.
- Compute uses 13B, memory holds 47B.
- Each layer holds 8 expert blocks and a router picks 2 of them per token.
- Only 13B of the 47B parameters run per token.
- All 47B parameters must stay in memory.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Start from the two published figures: about 47B total parameters and 13B active.1 The paper's abstract gives both. Call the shared parameters s and the parameters of one expert, summed over all layers, e. Then s + 8e = 47 and s + 2e = 13, in billions. Subtracting gives 6e = 34, so e is about 5.67B and s is about 1.67B (illustrative, because the published figures are rounded).1 About 96.5% of the weights then sit inside expert blocks.
| Quantity | Value |
|---|---|
| Naive guess, 8 x 7B | 56B |
| Total parameters, paper | 47B |
| Active parameters per token, paper | 13B |
| Active share of the total | 27.7%1 |
| Shared parameters (illustrative split) | 1.67B1 |
| One expert across all layers (illustrative) | 5.67B1 |
| Weights at fp16, 2 bytes each | 94 gigabytes1 |
| Weights at 8-bit, 1 byte each | 47 gigabytes1 |
| Weights at 4-bit, 0.5 byte each | 23.5 gigabytes1 |
The byte counts cover weights only. They leave out the activations and the memory for the running context, which add more.
Text description of the diagram
- At 2 bytes per parameter, 47B parameters take 94 GB.
- At 8-bit the weights take 47 GB and at 4-bit they take 23.5 GB.
Trade-offs and edge cases
A sparse model trades memory for compute. The paper reports that Mixtral outperforms or matches Llama 2 70B across all evaluated benchmarks, and vastly outperforms it on mathematics, code generation and multilingual tests, while using 13B active parameters per token. The price is that a serving stack needs the memory of a 47B model. When one GPU cannot hold it, the weights are split across devices; see data parallelism vs model parallelism. A common MoE choice is expert parallelism: the experts sit on different devices, and tokens are sent to the devices holding their chosen experts at every MoE layer, which adds communication cost (see the expert-parallelism and capacity-factor sections of the Hugging Face post).
Text description of the diagram
- A dense 13B model holds and uses 13B parameters.
- Mixtral holds 47B parameters but uses 13B per token.
- A dense 47B model holds and uses 47B parameters.
Three edge cases matter. First, load balance: if the router favours a few experts, they train faster and get picked more, so an auxiliary loss is added in MoE training generally to give all experts roughly equal use. With 1,000 tokens and 2 experts each there are 2,000 assignments, so a balanced router gives 250 per expert; a skewed one might give 600 to the busiest, 2.4 times the mean (illustrative).1
Second, small batches: tokens spread over experts, so batches per expert are uneven and hardware can sit idle, a capacity and communication issue the Hugging Face post discusses. Third, fine-tuning: the Hugging Face post says MoEs have historically struggled to generalize during fine-tuning, so overfitting is the risk to watch. If a smaller footprint is the goal, a small language model is the other route.
How has mixture-of-experts design changed since Mixtral?
Newer open models split the feed-forward work into many more, smaller experts. DeepSeek-V3 (DeepSeek-AI, December 2024) uses 1 shared expert, which every token always passes through, plus 256 routed experts per MoE layer, and it activates 8 routed experts per token. It has 671B total parameters against 37B active per token. Mixtral, by contrast, routes each token to 2 of 8 equally sized experts and has no shared expert.
The core trade is unchanged: compute follows the active parameters, while memory follows the total, so all 671B parameters must still be held. Fine-grained routing mainly gives the router many more combinations of experts to choose from for each token. This is one documented example, not a claim about every later model.
How to answer it in an interview
What is Mixtral 8x7B, and why is it cheaper to run than a dense model of the same size?
Mixtral 8x7B is a sparse mixture-of-experts transformer. Each layer has 8 feed-forward experts and a router that picks 2 per token, at every layer. Attention and embeddings are shared, so the total is about 47B parameters, not 56B, and only about 13B are active per token. Compute therefore looks like a 13B model. But every expert must sit in GPU memory, because the next token may need any of them, so memory follows the 47B. Fine-tuning needs care because MoEs can overfit.
The spoken structure: name it, say 8 experts and 2 per token, give 47B against 13B, then state that compute is cheap and memory is not.
Common mistakes
- "Eight experts of 7B each means 56B parameters." (wrong) Only the feed-forward blocks are copied, and the rest is shared. (right) "The total is about 47B."
- "Only 13B are active, so it fits in 13B worth of memory." (wrong) The next token may route anywhere. (right) "Compute follows 13B, memory follows 47B."
- "The router picks two experts for the whole prompt." (wrong) Routing happens per token and per layer. (right) "Each token picks its own two experts at every layer."
- "Mixtral is eight separate 7B models, one per topic." (wrong) The experts are blocks inside one network, trained jointly with the router. (right) "One model, with 8 experts in every layer."
Follow-up questions
Is Mixtral 8x7B open source?
The paper states that both the base and instruct models are released under the Apache 2.0 license.
What context length does Mixtral 8x7B use?
The paper's abstract says the model was trained with a context size of 32k tokens. Longer contexts also grow the memory needed beyond the weights.
How much memory does Mixtral 8x7B need, and how is it served across several GPUs?
For the weights alone, 94 GB at fp16, 47 GB at 8-bit and 23.5 GB at 4-bit, because all 47B parameters stay loaded.1 When the weights do not fit on one card, the experts can be placed on different devices (expert parallelism). Tokens then travel to the devices holding their chosen experts at each MoE layer, so communication cost joins the memory bill, as the Hugging Face post explains.
How does Mixtral 8x7B compare with Llama 2 70B?
The paper reports that it outperforms or matches Llama 2 70B on all evaluated benchmarks, using 13B active parameters per token. Memory is lower too, 47B against 70B weights, though not by as much as the active count suggests.
What stops the router from sending every token to the same experts?
In MoE training generally, an auxiliary loss encourages equal importance across experts, which breaks the loop where favoured experts train faster and get picked more.
Can you fine-tune Mixtral 8x7B?
Yes: the authors also provide an instruction-following version, Mixtral 8x7B Instruct. For your own data, parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) train small added matrices, but the full 47B weights still need loading. See LLM fine-tuning memory, and check fine-tuning vs prompt engineering first.
- Mixtral 8x7B has 8 feed-forward experts per layer, and a router picks 2 per token.
- Only the feed-forward blocks are copied, so the total is about 47B, not 56B.
- Active parameters are about 13B per token, so compute resembles a 13B model.
- Every expert stays in memory, so the memory bill follows the 47B.
- Fine-tuning needs care: MoEs can overfit, and a load-balancing loss keeps the experts in use.
Sources
- Mixtral of Experts (arXiv 2401.04088v1) — arXiv (Jiang et al., Mistral AI), 8 Jan 2024
- Mixture of Experts Explained — Hugging Face, 11 Dec 2023
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv 1701.06538) — arXiv (Shazeer et al.), 23 Jan 2017
- DeepSeek-V3 Technical Report (arXiv 2412.19437v1) — arXiv (DeepSeek-AI), 27 Dec 2024
