Transformers & Attention

Why does attention require scaling?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

Why does attention require scaling? Because query-key dot products grow with the key dimension d_k, pushing the softmax into a saturated region with tiny gradients; dividing the scores by √d_k keeps them stable. The follow-up interviewers ask next is why the factor is a square root rather than d_k itself, and the answer is one line of variance arithmetic: the dot product has variance d_k.

In this article

Before you start
Why interviewers ask this
Attention scaling from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one step of the transformer architecture learning path.

Why interviewers ask this

Without the scaling term, attention trains badly once the key dimension is large, so this question separates a candidate who knows the mechanism from one who has memorized the formula. A weak answer says "it normalizes the scores" and cannot say what is being normalized. A strong answer names the three links in the chain: the dot-product variance grows with d_k, a large score saturates the softmax, and a saturated softmax passes a very small gradient. It also says why the factor is a square root.

A Staff-level answer adds the caveats. The variance argument assumes idealized components, and the original authors only say they suspect the effect. The scaling is a fixed constant with no learned parameters. It can be read as a fixed softmax temperature. Interviewers often follow with "what if you divided by d_k instead?", which the steps below answer with numbers.

Attention scaling from first principles

Attention scaling factor: The attention scaling factor is a fixed constant, one divided by the square root of the key dimension d_k, that multiplies every query-key dot product before the softmax in scaled dot-product attention. It adds no learned parameters, and its job is to keep the spread of the scores the same whatever the length of the query and key vectors.
Softmax saturation: Softmax saturation is the state in which one input to a softmax is so much larger than the others that the output is almost one-hot, with nearly all the weight on one entry and nearly none on the rest. In that state the softmax's gradient is close to zero, so the layers before it receive almost no learning signal.

In plain words

Picture a walker who takes one random step per vector component. After many steps the walker is typically far from the start, and more steps mean a bigger typical distance. A dot product is like that walk: each component adds one random push. The analogy breaks in one place. The pushes here are products of two numbers, not coin flips, but the same law decides how far the total drifts.

Now picture the softmax as a contrast knob. When scores are close, the weights are shared out. When one score towers over the rest, that entry takes everything.

In one precise sentence, scaling divides each score by the standard deviation the dot product would have for random inputs, so the softmax always sees scores of a similar size.

Large keys give large scores, a saturated softmax and tiny gradients, and one fixed division removes the cause.CC BY 4.0 — source: Why does attention require scaling?

Step-by-step description

The dot product has variance d_k.
One key takes nearly all the weight.
The softmax slope is nearly zero.
Dividing by √d_k restores variance one.

The dot product of two random vectors has variance d_k.
Large scores saturate the softmax, leaving extremely small gradients.
Dividing by √d_k returns the variance to one.

Step 1: Dot products grow with the key dimension

A query-key dot product is a sum of $d_{k}$ products, so its spread grows with $d_{k}$ . Variance is the average squared distance of a value from its mean, and the standard deviation is the square root of the variance. Vaswani et al. (2017) assume the components of q and k are independent random variables with mean zero and variance one. Then the dot product has mean zero and variance $d_{k}$ .

The reason is short. Each product of two independent unit-variance numbers has mean zero and variance one. Variances of independent terms add, so $d_{k}$ terms give variance $d_{k}$ :

Var (q \cdot k) = \sum_{i = 1}^{d_{k}} Var (q_{i} k_{i}) = d_{k}

The standard deviation is therefore $\sqrt{d_{k}}$ , which is the square root the interviewer asks about. It is eight at a d_k of 64 and 32 at a d_k of 1,024.¹ In twenty thousand seeded random trials per size, the measured variance stayed within five percent of d_k at every size from 4 to 1,024.¹

You now know the cause: longer vectors produce scores with a wider spread.

The spread of an unscaled score grows with the square root of the key dimension.CC BY 4.0 — source: Why does attention require scaling?

Text description of the diagram

The standard deviation of an unscaled score is the square root of d_k.
It is eight at a d_k of 64 and 32 at a d_k of 1,024.

Step 2: Large scores saturate the softmax

A softmax saturates when its largest score exceeds the others by a wide margin, because the softmax exponentiates its inputs. Take one row of four scores, eight, zero, minus eight and four. This row is illustrative: it spreads about one standard deviation each way for a d_k of 64. A gap of eight multiplies a key's weight by about 2,981.¹

The softmax weights of that row are about 0.98 on the first key, 0.018 on the last and under 0.0004 on the other two.¹ The row is close to one-hot, so the model attends to a single token whatever the other keys contain.

Vaswani et al. (2017) describe the effect this way: they suspect large dot products push "the softmax function into regions where it has extremely small gradients". Large scores saturate the softmax; the next step shows what that does to learning.

Step 3: A saturated softmax passes almost no gradient

A saturated softmax passes almost no gradient because the softmax's slope shrinks as its output approaches one-hot. The Jacobian of the softmax is the table of partial derivatives that shows how each output weight changes when each input score changes. For weights p it equals the diagonal matrix of p minus the outer product of p with itself. Its diagonal entries are p times one minus p.

For a weight near one or near zero, that product is near zero. In the saturated row of Step 2, the largest entry of the Jacobian is about 0.018.¹ After dividing the scores by √d_k, the largest entry is about 0.2493, roughly fourteen times larger.¹

During training, the loss signal travels back through this Jacobian. If it is tiny, the scores for keys that deserve more weight barely move. Training slows or stalls even though the model is fine. Scaling attacks the cause, the score size, instead of the symptom.

Step 4: Dividing by √d_k restores a stable spread

Dividing each score by √d_k returns the variance of the dot product to one. The standard deviation √d_k divided by √d_k is one. The factor is a fixed constant, so it adds no learned parameters and almost no compute. In the paper's words, "we scale the dot products by" the reciprocal of √d_k. The full formula puts the factor inside the softmax:

Attention (Q, K, V) = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}}) V

Under the paper's independence assumption, dividing by $d_{k}$ instead overshoots. The variance becomes one over $d_{k}$ , and the weights go almost flat. The illustrative row shows it: for the example row the weights are about 0.28, 0.25, 0.22 and 0.26.¹ With √d_k they are about 0.47, 0.17, 0.06 and 0.29: still peaked, but alive.¹

Dot-product attention is worth keeping stable because the paper reports it is "much faster and more space-efficient in practice", since it uses optimized matrix multiplication. The same paper says additive attention "outperforms dot product attention without scaling for larger values of d_k". Scaling is the paper's remedy for that gap.

Without scaling one key takes nearly all the weight; dividing by d_k instead leaves the weights almost flat.CC BY 4.0 — source: Why does attention require scaling?

Text description of the diagram

Without scaling, the first key takes about 0.98 of the attention weight.
Dividing by d_k instead leaves the weights almost flat.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.