How does Mixtral work?

In this article

1. Before you start
2. Why interviewers ask this
3. Mixtral from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Mixtral is one of the model designs behind how large language models are trained.

Why interviewers ask this

Questions about mixture-of-experts models are reported by candidates in large language model interviews, because the topic tests whether you can separate compute from memory.

A weak answer says "Mixtral is eight 7B models and a voter." That is wrong on two counts: the experts are not whole models, and the total is not 56 billion parameters. A strong answer describes the router, the top-two choice and the weighted sum, per token and per layer. A Staff-level answer adds the costs: all experts must be in memory, small batches use hardware badly, and fine-tuning is harder. The Mixtral 8x7B overview covers the model in general.

Mixtral from first principles

Expert

An expert is a feed-forward block that is one of several parallel blocks inside a layer, each with its own weights; the router sends each token to only some of them, and the other experts do no work for that token.

In plain words

Picture a hospital front desk. Every patient (a token) is seen by a triage nurse (the router), who sends them to two of eight specialist rooms (the experts). The patient's results are then blended, with more weight on the room the nurse trusted more. The hospital owns all eight rooms all the time, but each patient uses only two. The analogy breaks in one place: Mixtral's rooms are not medical specialties. Nobody assigned them topics; the router learned its own split.

In one precise sentence, replacing one block with several parallel blocks and a chooser is called a mixture of experts (MoE), and Mixtral applies it to the feed-forward block of every layer while keeping attention shared (Jiang et al., 2024).

Why small compute still needs big memory

Mixtral computes like a small model but needs the memory of a large one, because the router chooses experts per token and per layer, so any expert may be needed next. Across a batch of tokens and across the 32 layers, different tokens take different paths, so a server cannot know in advance which two experts it will need and keeps all eight loaded in every layer.

The paper says so directly: "the memory costs for serving Mixtral are proportional to its sparse parameter count, 47B" (Jiang et al., 2024, §3), and the Hugging Face explainer notes that all parameters must be loaded. At 2 bytes per parameter, the weights alone take 93.4 GB.¹ All eight experts stay in memory even though only two run. The escape hatch is moving experts out to the host computer's main memory, which frees accelerator memory but can make each token wait on transfers. Step 4 gives the exact parameter counts.

Step 1: A router scores all eight experts for each token

A router turns one token's vector into one score per expert. In Mixtral the token vector has 4,096 numbers and there are 8 experts (Jiang et al., 2024, Table 1), so the router is a matrix that maps 4,096 numbers to 8 scores, called logits. The router scores all eight experts for each token, in every layer, and these scores alone decide which experts run.

Input: one token vector. Output: eight numbers. The paper states that the expert layer is applied independently per token and replaces the feed-forward block of the transformer layer. So the router never judges the whole prompt; it judges each token separately. You now know where the choice is made; the next step is how the eight scores become a decision.

Step 2: The top two scores are kept and turned into weights

Top-2 routing is the rule that keeps the two largest router scores, discards the other six and normalizes the two survivors with a softmax. Only the two highest-scoring experts run. The paper writes it as the softmax of the top-K of the scores, with K set to 2 for Mixtral; the discarded scores are set to minus infinity, so their weight is exactly zero (Jiang et al., 2024, §2.1).

Take illustrative logits of 1.2, −0.5, 3.1, 0.4, 2.2, −1.0, 0.0 and 0.9 for experts 1 to 8. Experts 3 and 5 score highest, so they are kept. The softmax over just those two gives weights of 0.7109 and 0.2891.¹ Input: eight logits. Output: two expert numbers and two weights that sum to one. The number K is a hyper-parameter that sets how much compute each token receives.

Step 3: The two experts' outputs are added with softmax weights

Each chosen expert transforms the token vector, and the layer's output is the weighted sum of the two results. In symbols, with weights $g_i$ from Step 2 and expert functions $E_i$:

Each expert has the same shape as the feed-forward block of Mistral 7B, which is the base architecture Mixtral extends (Jiang et al., 2024). With illustrative expert outputs of 2.0 and 0.5 and the weights above, the layer output is 0.7109 × 2.0 + 0.2891 × 0.5 = 1.5664.¹

The same routing then repeats. The model has 32 layers (Jiang et al., 2024, Table 1), and each layer has its own router, so the same token can meet different experts in different layers, and different tokens can take different paths. Attention stays shared and dense: it uses 32 query heads that share 8 key-value heads, a design called grouped-query attention, and the context length is 32,768 tokens (same table). Input: a token vector. Output: a vector of the same size, ready for the next layer.

Step 4: Total parameters set memory, active parameters set compute

Mixtral's parameters split into a large stored total and a small active part. Mistral reports 46.7B total parameters but only 12.9B used per token (Mistral AI, 2023). A model that runs every parameter on every token is called a dense model; Mixtral is not dense. Compute per token follows the active count, which is about 27.6% of the total.¹ Memory follows the total, for the reasons in the previous section. So Mixtral computes like a model of about 13B parameters and stores like one of about 47B.

1 · Score2 · Pick two3 · Mix4 · MemoryReplay

Step-by-step description

1. The router scores eight experts.
2. Only the top two experts run.
3. Outputs are added with softmax weights.
4. All eight experts stay loaded.

• The router scores all eight experts for each token.
• Only the two highest-scoring experts run.
• All eight experts stay in memory even though only two run.

Text description of the diagram

• Mixtral has 46.7B total parameters and uses 12.9B per token.
• Llama 2 70B uses all 70B parameters for every token.

The chart shows Mixtral with 46.7B total and 12.9B active parameters per token, against 70B used for every token by Llama 2 70B (Jiang et al., 2024, Table 2). Dividing 70B by the paper's rounded 13B active gives about 5.4 times fewer active parameters.¹

Where do 46.7 billion and 12.9 billion come from?

The two totals follow from the architecture table by arithmetic. Eight copies of a 7B-sized model would be 56B, but only the feed-forward blocks are replicated and the rest is shared (Hugging Face, 2023). Each expert is a SwiGLU feed-forward block (Jiang et al., 2024, §2.1), which has three weight matrices of 4,096 by 14,336 (Table 1). One expert then holds 176,160,768 parameters, and the 256 experts (8 in each of 32 layers) hold 45.10B.¹ Attention, embeddings, router and norms add 1.61B, so the total is 46.70B, matching the reported 46.7B.¹ Running two experts per layer gives 11.27B plus 1.61B, or 12.88B, matching the reported 12.9B.¹ The matrix sizes come from the paper; the totals are our arithmetic.

The Staff-level answer

The spoken structure: say what is replaced, say how two experts are chosen per token and layer, then split compute from memory and close with the cost. What makes it Staff-level is the last two sentences: it separates active from total parameters and names what the design costs.

The rubric

Three variants of this question

What does top-2 routing do in Mixtral?

Top-2 routing sends each token through only the two experts with the highest router scores and mixes them with softmax weights. The other six experts do no work for that token at that layer, which is what keeps compute per token near that of a much smaller dense model.

Does each expert in Mixtral specialize in a topic?

Mixtral's experts do not specialize by topic: the authors report that they do not observe obvious topic patterns in expert assignment (Jiang et al., 2024). The routing looks more syntactic, with some regularity in tokens such as indentation in code. Consecutive tokens also repeat the same first-choice expert more often than chance: random assignment would give 1 in 8, or 12.5%, and the paper's Table 5 reports first-choice repeats of roughly 24% to 28% at layer 15.

How much graphics processing unit (GPU) memory does Mixtral need if you serve it?

Weights alone are about 93.4 GB at 16-bit precision, because all 46.7B parameters are stored.¹ A 12.9B dense model would need about 25.8 GB, so the savings are in compute, not memory. Lower-precision numbers shrink the figure, but the ratio between total and active stays the same.

Common mistakes

• "Mixtral is eight 7B models, 56B parameters." (wrong) Only the feed-forward blocks are replicated; the rest is shared, so the total is 46.7B. (right) "Eight experts per layer, shared attention, about 47B total."
• "It only uses 12.9B parameters, so it fits on a small graphics processing unit (GPU)." (wrong) Every expert must be loaded. (right) "Compute follows the active parameters; memory follows the total."
• "The router picks one expert for the whole prompt." (wrong) Routing is per token and per layer, two experts each time. (right) "Two experts per token, chosen again in each layer."
• "One expert handles math, another handles code." (wrong) The paper finds no obvious topic pattern. (right) "Routing looks syntactic and has some locality across neighboring tokens."

Follow-up questions

How does the router in Mixtral avoid using only a few experts?

Mixture-of-experts training tends to collapse onto a few favored experts, so such models add an auxiliary loss that encourages equal use of all experts (Hugging Face, 2023). The Mixtral paper does not describe its own balancing loss, so do not claim details of it. Experts and routers are trained jointly on web data (Mistral AI, 2023).

Is Mixtral really six times faster than Llama 2 70B?

Mistral claims 6x faster inference than Llama 2 70B and about the speed and cost of a 12.9B model, but the post gives no method (Mistral AI, 2023). The paper is more careful: expert layers suit batched workloads, with routing overhead and lower hardware use at small batch sizes. Splitting experts across devices is a form of the model parallelism described in data versus model parallelism.

Is Mixtral harder to fine-tune than a dense model?

Historically yes: mixture-of-experts models have struggled to generalize in fine-tuning and tended to overfit (Hugging Face, 2023). The same explainer calls recent instruction-tuning work on mixture-of-experts models promising, and Mixtral Instruct itself was tuned successfully with supervised fine-tuning and direct preference optimization (Jiang et al., 2024). General fine-tuning mechanics and memory costs are in fine-tuning a large language model (LLM), and adapter methods are in low-rank adaptation (LoRA) fine-tuning.

How was Mixtral Instruct trained?

The instruction version was trained with supervised fine-tuning (SFT) on an instruction dataset, followed by direct preference optimization (DPO) (Jiang et al., 2024). The two alignment methods are compared in DPO versus reinforcement learning from human feedback (RLHF).

Is Mixtral better than Llama 2 70B?

On the paper's Table 2, Mixtral scores 74.4% against 69.6% on a grade-school math set and 40.2% against 29.3% on HumanEval, a code benchmark, with 13B active parameters against 70B (Jiang et al., 2024). Read these as the authors' reported results, not a universal ranking.

Sources

1. Mixtral of Experts (arXiv 2401.04088) — arXiv (Jiang et al., Mistral AI), 8 Jan 2024
2. Mixtral of experts — Mistral AI, 11 Dec 2023
3. Mixture of Experts Explained — Hugging Face, 11 Dec 2023