In this article
  1. Before you start
  2. Why interviewers ask this
  3. Self-attention from first principles
  4. Worked example
  5. Trade-offs and edge cases
  6. How to answer it in an interview
  7. Common mistakes
  8. Follow-up questions
  9. Sources

Before you start

This page is one step of the transformer learning path.

Why interviewers ask this

Self-attention is the operation every transformer layer runs, so a candidate who cannot describe it step by step has not understood the architecture most language-model teams build on. The interviewer wants to hear where queries, keys and values come from, why the scores are rescaled, and what changes when the model must not look ahead.

A weak answer says "attention lets the model focus on important words." A strong answer names the three vectors, writes the formula, says that each row of weights sums to one, and can say what each of the three attention kinds in a transformer reads. A Staff-level answer adds the cost: the score matrix grows with the square of the sequence length, which is why long contexts need special handling.

Self-attention from first principles

Query, key and value
A query, key and value are three vectors that a transformer layer computes for every token by multiplying the token's vector with three separate learned matrices; the query asks what to look for, the key says what the token offers, and the value is what gets passed on.
Attention weight
An attention weight is a number between zero and one that says how much one token's value vector contributes to another token's output; the weights in one row sum to one.
Encoder and decoder
An encoder is a stack of transformer layers that reads a whole input sequence at once; a decoder is a stack that produces output one token at a time, using the causal mask so it sees only earlier tokens. A decoder-only transformer has decoder layers and no encoder.

In plain words

Picture a meeting. Each person writes a question on a card (the query) and wears a name tag that says what they know (the key). Each also holds a folder of notes (the value). Everyone compares their question with every name tag. They decide how much each other person matters. Then they build a summary from the folders, giving more room to the best matches. The analogy breaks in one place. The questions, tags and folders are learned lists of numbers, and the matching is arithmetic, not judgement.

In one precise sentence, self-attention replaces each token's vector with the weighted sum of all value vectors. The weight of token j for token i is a softmax over the scaled dot products of the query of i with every key. Vaswani et al. (2017) used it for translation. It builds on Bahdanau et al. (2014), who let a translation model search over the source words that matter for each target word.

Step 1: Each token becomes a query, a key and a value

Every token vector is multiplied by three learned matrices to give three new vectors: a query, a key and a value. The matrices are the same for every token in a layer, so the same rule is applied to every position, and each layer has its own matrices.

Inputs: one vector per token. Outputs: three vectors per token, all of the same small size called the key size, d_k. The step does not look at any other token, so it can run for all tokens at the same time on a graphics processing unit (GPU). You now know where the three roles come from: they are three different views of one token, made by learned multiplication, as Alammar (2018) shows in pictures. The word "self" refers to this: all three views are made from the same sequence.

In practice a layer runs this recipe several times side by side, and each run is an attention head: a head is one independent set of query, key and value matrices with its own table of weights. The outputs of all heads are joined end to end and multiplied by a fourth learned matrix. The output matrix is that fourth matrix, and it returns them to the token vector's original length; multi-head attention covers this in full. The toy example below uses one head.

Step 2: Every query is scored against every key

The score between token i and token j is the dot product of the query of i with the key of j. A large score means that j offers what i is looking for. Doing this for every pair fills a square table of scores with one row per query and one column per key.

Inputs: all queries and all keys. Output: an n-by-n table of raw scores for a sequence of n tokens. The table is why attention connects far-apart words in one step: the first and the last token are compared directly, without passing through the words between them. It is also why attention is costly, because the table has n times n entries. You now know what the raw scores are and why there are n squared of them.

Step 3: Scale, mask and softmax turn scores into weights

Raw scores become attention weights in three moves. First each score is divided by the square root of the key size, which keeps the numbers from growing with the vector size. The reason is that a dot product of two vectors with d_k independent entries of mean 0 and variance 1 has variance d_k, so for large d_k some scores become very large, and Vaswani et al. suspected that softmax then sits where its gradients are extremely small (Vaswani et al., 2017, §3.2.1, note 4).

Dividing by the square root of d_k brings the variance back to 1. Second, in a decoder, every score above the diagonal is replaced by minus infinity, which is the causal mask. Third, softmax is applied to each row, turning it into positive weights that sum to one.

The mask is what answers the follow-up question. After softmax, a hidden position gets weight exactly zero, so the first token can attend only to itself and its weight on itself becomes 1.0. In the worked example below the mask turns token A's weights 0.0454, 0.7679 and 0.1867 into 1.0, 0 and 0.1 PyTorch documents the same recipe: its scaled dot-product attention function computes softmax of the query-key product times a scale, and its causal option applies a lower-triangular mask.

Step 4: The weights mix the value vectors

Each token's output is the sum of all value vectors, each multiplied by that token's attention weight for it. A token with a weight of 0.7679 (illustrative) on another token takes mostly that token's value; a row with equal weights takes the plain average.

Inputs: the table of weights and the value vectors. Output: one new vector per token, the same size as the value vectors. With the queries, keys and values stacked as the rows of matrices Q, K and V, the four steps are equation 1 of Vaswani et al. (2017), §3.2.1:

Attention(Q,K,V)=softmax(QKTdk)V

The superscript T turns K on its side, so Q times it is the whole score table of Step 2; a decoder adds the causal mask to that table before the softmax. The outputs go on to the rest of the layer, and every layer repeats this. This is the sense in which attention mixes information between tokens, while the feed-forward network that follows works on each token separately; the transformer overview shows where this sits in a layer.

Self-attention scores every pair of tokens, then mixes valuesFlow of one self-attention layer: token vectors become queries, keys and values, queries are scored against keys, a scaled softmax turns scores into weights, and the weights mix the value vectors into each token's output.Token vectorsone per positionQ, K, Vthree learned mapsQ and KScoresq dot k, all pairsWeightsscale, mask, softmaxtimes VOutputsweighted sum of VDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/self-attention.html

Every token's output is a weighted mix of all value vectors, with weights from query-key scores.CC BY 4.0 — source: What is self-attention, and how does it work step by step?
Step-by-step description
  1. Each token becomes a query, a key and a value.
  2. Each query is compared with every key.
  3. Scale, mask if needed, then softmax into weights.
  4. Weights blend the value vectors into outputs.
  • Each token gets a query, a key and a value vector from three learned matrices.
  • The score of a pair is the dot product of one query and one key.
  • Dividing by the square root of the key size and applying softmax gives weights that sum to one.
  • The output of a token is the weighted sum of the value vectors.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

A three-token toy with a key size of 2 can be done with a calculator. The embeddings and the three matrices are illustrative numbers chosen for the walkthrough, not taken from any model.1 Each token's embedding has four numbers, and multiplying it by the three matrices gives the queries, keys and values below. The three tokens are called A, B and C. Their queries are (2, 0), (0, 2) and (1, 1); their keys are (0, 2), (2, 0) and (1, 1); their values are (3, 2), (1, 2) and (1, 3). The raw scores are the dot products: token A scores 0, 4 and 2 against keys A, B and C.

Dividing by the square root of 2 gives 0, 2.8284 and 1.4142. Softmax raises e (about 2.7183) to each scaled score, giving 1, 16.9188 and 4.1133, and divides each by their total, 22.0321. That turns the row into weights of 0.0454, 0.7679 and 0.1867, which add to one.1 Token B mirrors it, and token C, whose query matches all keys equally, weighs all three at 0.3333.

In the toy example token A takes 0.7679 of its output from token B, and the same row under a causal mask gives token A weight 1.0 on itself.
QuantityValue (illustrative toy)
Raw scores of token A on A, B, C0, 4, 2
Scaled scores of token A0, 2.8284, 1.4142
Attention weights of token A0.0454, 0.7679, 0.1867
Output of token A(1.0908, 2.1867)
Weights of token A with a causal mask1.0, 0, 0
Output of token A with a causal mask(3.0, 2.0)
Output of token C, both cases(1.6667, 2.3333)

The output of token A is 0.0454 times (3, 2) plus 0.7679 times (1, 2) plus 0.1867 times (1, 3), which is (1.0908, 2.1867).1 With the mask, token A can see only itself, so its output is its own value (3.0, 2.0). Token C comes last, so the mask hides nothing from it, and its output is the same in both cases.

Each token spreads one unit of weight across all tokensGrouped bars of illustrative attention weights for three tokens A, B and C. Each query token's three weights sum to one; tokens A and B favour each other, and token C weighs all three equally.Attention weight00.51Token AToken BToken CQuery tokenon Aon Bon C0.04540.76790.33330.76790.04540.33330.18670.18670.3333Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/self-attention.html
Each row of weights sums to one, so a token trades attention between positions.CC BY 4.0 — source: What is self-attention, and how does it work step by step?
Text description of the diagram
  • Illustrative weights of token A on tokens A, B and C are 0.0454, 0.7679 and 0.1867.
  • Token C weighs all three tokens at 0.3333.
  • Every row of weights sums to one.

Real sizes are larger but follow the same arithmetic. In the base model of Vaswani et al. (2017), §3.2.2 the model width (the length of each token vector) is 512 and there are 8 heads, so each key size is 512 divided by 8, which is 64; the four matrices of one attention layer (query, key, value and output) hold 4 × 512 × 512 = 1,048,576 weights.1 The score table for a 1,024-token sequence has 1,048,576 entries per head; at 8,192 tokens it has 67,108,864, which is 64 times more for an 8-times longer sequence.1

Trade-offs and edge cases

Self-attention buys direct connections between all positions at the price of a score table that grows with the square of the sequence length. Doubling the length quadruples the table, which is why long-context work focuses on reading and writing that table efficiently; FlashAttention computes the same result without storing the whole table, and the key-value (KV) cache avoids recomputing keys and values during generation.

The three kinds of attention in a transformer differ only in where the inputs come from and what is hidden. Self-attention takes queries, keys and values from one sequence. Cross-attention is a variant in which queries come from one sequence, such as a decoder's, while keys and values come from another, such as an encoder's output; the original translation model used it between decoder and encoder (Vaswani et al., 2017). Masked self-attention is self-attention with the causal mask.

Self, cross and masked attention differ in where K and V come fromTable comparing self-attention, cross-attention and masked self-attention by where queries, keys and values come from, whether later positions are visible, and where each is typically used.QuestionSelfCrossMaskedselfQueries comefromthesequencethe decoderthesequenceKeys, valuesfromthe samesequenceencoderoutputthe samesequenceLater positionsvisiblevisiblehiddenTypical useencoderstranslationdecoderstextgenerationDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/self-attention.html
The three kinds differ only in the source of keys and values and in which positions are hidden.CC BY 4.0 — source: What is self-attention, and how does it work step by step?
Text description of the diagram
  • Self-attention takes queries, keys and values from one sequence.
  • Cross-attention takes queries from one sequence and keys and values from another.
  • Masked self-attention hides later positions so a decoder cannot read ahead.

Edge cases: attention alone has no sense of order, so a model must add position information; and attention weights are a description of how values were mixed in one layer, not proof of why the model produced an answer. The large language model (LLM) inference overview covers the serving side of these costs.

How to answer it in an interview

Interviewer asks

What is self-attention and how does it work?

A strong answer

Self-attention lets every token build its output from all tokens in the same sequence. Each token vector is projected into a query, a key and a value. I score every query against every key with a dot product, divide by the square root of the key size, apply softmax so each row of weights sums to one, and take the weighted sum of the value vectors. In a decoder I add a causal mask before softmax so a token cannot see later tokens. The cost is an n-by-n score table, so it grows with the square of the sequence length.

The spoken structure: say what it produces, name the three vectors, walk the four steps, then give the mask and the cost.

Common mistakes

  • "Attention picks the single most important word." (wrong) Softmax spreads weight across all tokens, and the output is a blend. (right) "It forms a weighted average, and the weights in each row sum to one."
  • "Queries, keys and values are three different inputs." (wrong) In self-attention all three are made from the same sequence by three different matrices. (right) "Three learned projections of the same token vectors."
  • "The mask changes the scores after softmax." (wrong) The mask is applied before softmax, so hidden positions end with weight exactly zero and the rest still sum to one. (right) "I set masked scores to minus infinity, then apply softmax."
  • "Self-attention is the same as cross-attention." (wrong) Cross-attention takes keys and values from a different sequence. (right) "Same recipe, different source of keys and values."

Follow-up questions

Why is self-attention called self?

The word "self" says that the queries, keys and values all come from the same sequence. In cross-attention the queries come from one sequence and the keys and values from another, so the sequence is attending to something other than itself.

What is the attention mechanism, and how is it different from self-attention?

The attention mechanism is the general idea of learning weights over a set of items and using them to mix information, first applied to translation by Bahdanau et al. (2014). Self-attention is the case in which the items and the thing doing the looking are the same sequence.

What is a causal attention mask?

A causal attention mask hides every later position from each token. In the toy example it turns the weights of token A from 0.0454, 0.7679 and 0.1867 into 1.0, 0 and 0, because A is first and can see only itself.1

What is self-attention in an LLM?

An LLM is usually a decoder-only transformer, a stack of decoder layers with no encoder, and every layer applies masked self-attention to the tokens of the prompt and the text produced so far. The transformer overview shows how the layers fit together, and the deep learning overview covers the training side.

Why does self-attention cost grow with the square of the sequence length?

Every token is scored against every token, so a sequence of n tokens needs n times n scores per head. Going from 1,024 to 8,192 tokens multiplies the table by 64.1

Key takeaways
  • Self-attention gives each token a weighted average of all value vectors in the same sequence.
  • Queries, keys and values are three learned projections of the same token vectors.
  • Scores are scaled, optionally masked, then softmaxed so each row of weights sums to one.
  • A causal mask sets hidden scores to minus infinity before softmax, so hidden weights are exactly zero.
  • The n-by-n score table makes cost grow with the square of the sequence length.

Sources

  1. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
  2. Neural Machine Translation by Jointly Learning to Align and Translate (arXiv 1409.0473) — arXiv (Bahdanau, Cho, Bengio), 1 Sep 2014
  3. torch.nn.functional.scaled_dot_product_attention, PyTorch 2.14 documentation — PyTorch (date = read date), 25 Oct 2026
  4. The Illustrated Transformer — Jay Alammar, 27 Jun 2018