In this article
Before you start
This page sits in the transformer architecture learning path and assumes the ideas from how neural networks work.
Why interviewers ask this
A candidate who recites the formula but cannot say what breaks without it has memorized a line, not understood attention. Interviewers use this question to tell the two apart. Candidates report versions of it in interviews at AI labs and large tech companies: why attention needs scaling, what happens without it, and why the divisor is the square root of the key size.
A weak answer says the scaling makes numbers smaller or "stabilizes training". A strong answer gives the cause (a dot product of d_k terms has variance d_k), the effect (a saturated softmax and vanishing gradients) and the fix. A Staff-level answer adds the assumption behind the variance claim and explains why the divisor is a square root. The rubric further down spells out the three levels.
Scaled dot-product attention from first principles
- Saturated softmax
- A saturated softmax is a softmax whose largest input sits so far above the others that it outputs a weight near one for that entry and near zero for every other entry, so a small change in any input barely changes the output.
- Softmax Jacobian
- The softmax Jacobian is the table of derivatives of every softmax output with respect to every input; the entry for output i and input j is p_i times (δ_ij − p_j), and training uses it to pass the error signal back through the softmax.
In plain words
Picture a game where d_k people each add or subtract one point at random. After all of them have played, the total typically sits about √d_k points from zero, not d_k points. A dot product behaves the same way: it adds d_k random-looking products. The analogy breaks in one place. Real queries and keys are learned numbers, not coin flips.
d_k is the number of entries in each query vector and each key vector. In one precise sentence, attention divides every query-key dot product by √d_k so that the scores keep variance one at any d_k. The whole operation, with matrices Q, K and V holding the queries, keys and values, is called scaled dot-product attention:
Step 1: A dot product of d_k random entries has variance d_k
The score between a query and a key is a sum of d_k products, and the variance of that sum grows in step with d_k. Take a query q and a key k whose entries are independent with mean zero and variance one. Each product then has mean zero and variance one, and variances of independent terms add, so the sum has variance d_k and standard deviation √d_k.
Vaswani et al. (2017) state this assumption when they explain the scaling, and their base model uses d_k = 64, so its unscaled scores have a standard deviation of 8.1 In a simulation of 20,000 random pairs, the measured variance is 4.00 at d_k = 4, 64.49 at d_k = 64 and 508.16 at d_k = 512.1
Input: two vectors. Output: one score whose typical size is √d_k. You now know the scores grow with dimension; Step 2 shows why that hurts.
Step 2: A saturated softmax passes almost no gradient
A softmax fed widely spread scores saturates, and a saturated softmax gives training almost no signal. When one score leads the rest by many units, its weight is near one and every other weight is near zero. Training then relies on the softmax Jacobian, and when the weights are one-hot, every entry of it is near zero.
Take four illustrative scores of 16, 8, 0 and −8, a plausible spread when the standard deviation is 8. The weights are 0.999665 and 0.000335, and the other two round to zero. The largest Jacobian entry is 0.000335, against a possible maximum of 0.25.1 In 4,000 random trials with 8 keys at d_k = 64, the unscaled softmax gave its top key a mean weight of 0.876, and above 0.99 in 37.3% of trials.1
The failure is vanishing gradients, not exploding ones. Vaswani et al. give it as their suspected reason for the scale, and a 2026 survey restates it: large scores concentrate the probability on the maximum and the gradients elsewhere approach zero.
Step 3: Dividing by sqrt(d_k) restores unit variance
Dividing every score by √d_k returns the variance to one at every key size, while, under the independent unit-variance assumption at initialization, dividing by d_k would overshoot. Dividing a number by c multiplies its variance by 1/c², so turning d_k into one needs c² = d_k, which means c = √d_k.
After scaling, the measured variances are 1.001, 1.008 and 0.992 at d_k = 4, 64 and 512. Dividing by d_k instead leaves 0.2503, 0.0157 and 0.0019, scores so small that the softmax is nearly flat and attention cannot focus.1 The assumption is not universal: maximal update parametrization (μP, Yang et al., 2022) uses 1/d_k on purpose, because trained queries and keys correlate and their dot products grow with d_k rather than √d_k.
The same four illustrative scores become 2, 1, 0 and −1, with weights 0.6439, 0.2369, 0.0871 and 0.0321 and a largest Jacobian entry of 0.2293.1
Step-by-step description
- The query-key dot product has variance d_k.
- Wide scores make one weight near one and the gradient near zero.
- The divisor restores variance one before the softmax.
- A dot product of two d_k-dimensional vectors has variance d_k.
- A softmax fed widely spread scores saturates and passes almost no gradient.
- Dividing by sqrt(d_k) returns the score variance to one.
| d_k | Std of raw score | Largest weight, raw | Largest weight, scaled | Jacobian size, raw | Jacobian size, scaled |
|---|---|---|---|---|---|
| 4 | 2.00 | 0.530 | 0.346 | 0.339 | 0.360 |
| 64 | 8.03 | 0.876 | 0.363 | 0.151 | 0.366 |
| 512 | 22.54 | 0.958 | 0.364 | 0.057 | 0.365 |
Here "Jacobian size" is the square root of the sum of the squared Jacobian entries, averaged over rows. The "Std of raw score" column comes from the separate run of 20,000 random pairs, not from the 4,000 rows of 8 keys behind the other columns.
Text description of the diagram
- Without scaling the mean largest softmax weight is 0.53 at d_k 4 and 0.958 at d_k 512.
- With scaling the mean largest softmax weight stays near 0.36 at every d_k.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
Why do transformers scale attention scores by 1/sqrt(d_k), and what breaks if you skip it?
Each score is a dot product of a query and a key with d_k entries. If those entries have mean zero and variance one, the score has variance d_k, so its standard deviation is √d_k. Scores that spread out make the softmax nearly one-hot, and a one-hot softmax has a Jacobian near zero, so the query and key projections get almost no gradient. Dividing by √d_k, not d_k, brings the variance back to one at any dimension. The caveat is that trained vectors do not keep unit variance, so this is a sound default at initialization, not a proof.
What makes it Staff-level: it names the vanishing Jacobian rather than "big numbers", derives the square root from the variance arithmetic, and states the assumption.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says the scaling makes numbers smaller. | Says it keeps score variance at one as d_k grows. | Adds the assumption: independent, zero-mean, unit-variance entries. |
| Depth | Stops at "stability". | Links large scores to a saturated softmax. | Names the softmax Jacobian going to zero and the starved query and key gradients. |
| Trade-offs | Cannot say why a square root. | Explains that variances add, so the standard deviation grows as √d_k. | Notes that under the assumption dividing by d_k flattens attention, that trained vectors drift from unit variance, and that μP deliberately uses 1/d_k. |
| Communication | Recites the formula. | States the cause, then the effect, then the fix. | Gives one concrete number for the spread, then the consequence. |
| Follow-up readiness | Confuses vanishing and exploding gradients. | Corrects that confusion unprompted. | Can sketch the Jacobian and say what changes when d_k is small. |
Three variants of this question
What breaks in attention if you skip the 1/sqrt(d_k) scaling?
The model still runs but trains badly. At d_k = 64 the softmax saturates in 37.3% of simulated rows (top weight above 0.99), so the Jacobian shrinks to about 0.151 against 0.366 with scaling, and the query and key projections receive a weak signal.1
Does the scaling still matter when d_k is small?
Much less. At d_k = 4 the unscaled softmax already gives a mean top weight of 0.530 and a Jacobian size of 0.339, close to the scaled 0.346 and 0.360.1 The gap opens as d_k grows.
Do trained queries and keys still have unit variance?
Not exactly. The derivation assumes independent, zero-mean, unit-variance entries, which at best holds roughly at initialization. After training, entries correlate and their scale drifts, so the factor is a principled default and not a proof that scores stay at variance one. Modern models often fix logit growth directly: query-key (QK) normalization, which layer-normalizes queries and keys before the dot product, is used in ViT-22B, whose authors (Dehghani et al., 2023) report attention-logit divergence without it.
Common mistakes
- "It makes the numbers smaller, so training is more stable." (wrong) Smaller is not the goal; constant spread across dimensions is. (right) "It keeps score variance at one whatever d_k is, so the softmax does not saturate."
- "It stops exploding gradients." (wrong) The problem is vanishing gradients through a saturated softmax. (right) "Without it the softmax Jacobian goes to zero and the gradient vanishes."
- "We divide by d_k." (wrong) Under the independent unit-variance assumption at initialization that leaves variance 1/d_k and makes attention almost uniform; μP is a deliberate exception. (right) "We divide by √d_k, because dividing by c scales variance by 1/c² and we need c² = d_k."
- "After scaling the scores always have variance one." (wrong) That holds only under the independence and unit-variance assumption. (right) "It is exact at initialization under that assumption and approximate afterwards."
Follow-up questions
Why do you use the scaling factor (1/sqrt(d_k)) in attention?
Because a query-key dot product of d_k independent, zero-mean, unit-variance terms has variance d_k, and dividing by √d_k returns it to one so the softmax does not saturate; Step 1 and Step 2 give the numbers.
Does scaling prevent exploding gradients?
No. The scale targets the vanishing gradient of a saturated softmax; as the Step 2 numbers show, the Jacobian shrinks without it, it does not grow.
Is the scaling factor the same in multi-head attention?
The factor uses the key size of one head, not the model width. The original base model has eight heads with d_k = 64 each, per Vaswani et al., so each head divides by 8; see multi-head attention.
What did the original paper compare dot-product attention against?
Additive attention, which scores a pair with a feed-forward network with one hidden layer. Vaswani et al. (2017, §3.2.1), citing earlier experiments by Britz et al. (2017), note that the two perform similarly at small d_k but additive attention wins at larger d_k without scaling. They suspect large dot products push the softmax into regions with extremely small gradients.
Does the scaling change the cost of attention?
No. It adds one multiplication per score, so the quadratic cost stays; see why attention is quadratic and how to make it cheaper and the memory-efficient kernel FlashAttention.
Which transformer questions usually come next?
Interviewers often move to multi-head attention, positional information or attention cost. Practice them with interview questions on large language models.
- A dot product of d_k unit-variance entries has variance d_k and standard deviation √d_k.
- Large scores saturate the softmax, and a saturated softmax has a Jacobian near zero: the gradient vanishes.
- Dividing by √d_k restores variance one; under the same assumption dividing by d_k would flatten attention.
- The argument assumes independent, zero-mean, unit-variance entries, so it holds exactly under that assumption, which is only roughly true at initialization.
- The effect grows with d_k and is mild at d_k = 4.
Sources
- Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
- Attention mechanisms in neural networks (arXiv 2601.03329), survey — arXiv, 1 Jan 2026
- Massive Exploration of Neural Machine Translation Architectures (arXiv 1703.03906) — arXiv (Britz et al.), 11 Mar 2017
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (arXiv 2203.03466) — arXiv (Yang et al.), 7 Mar 2022
- Scaling Vision Transformers to 22 Billion Parameters (arXiv 2302.05442) — arXiv (Dehghani et al.), 10 Feb 2023
