Cross-attention vs self-attention, and what the causal mask does

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A mask that leaks the future makes a language model look brilliant in training and useless in use, and that bug is why this question is asked. Interviewers ask "where do Q, K and V come from in each block?", "implement causal masking in PyTorch" and "why minus infinity and not zero?". All three test one idea: attention is one formula, and the three blocks of a transformer differ only in where its inputs come from and which positions may look at which.

A weak answer describes three different mechanisms. The machine learning (ML) coding hub lists the other from-scratch rounds: ML coding interview questions. A strong answer says one formula, shows the source of the queries and of the keys and values in each block, and explains why the mask is applied before the softmax.

What each one is

Cross-attention

Cross-attention is an attention layer in which the queries come from one sequence, such as the decoder's, and the keys and values come from a different sequence, such as the encoder's output, so one sequence can read another.

Decoder-only model

A decoder-only model is a language model whose attention layers are all masked (causal) self-attention, with no encoder and no cross-attention, that generates text one token at a time from left to right.

Self-attention in plain words

Self-attention is a form of attention in which every token in a sequence looks at the other tokens in the same sequence and mixes in what it finds. A token is a word piece, the unit a model reads. Each token becomes three vectors: a query (what it seeks), a key (what it offers) and a value (what it passes on if chosen). A vector is a list of numbers; matrix multiplications turn each token's embedding into its three.

The attention score between two tokens is the dot product (multiply matching numbers, then add) of one's query with the other's key; softmax turns each row of scores into weights, and the output is the weighted sum of values. In the original transformer paper (Vaswani et al., 2017, section 3.2.3), in encoder self-attention "all of the keys, values and queries come from the same place", the previous encoder layer. Picture a meeting where each person weighs every speaker by relevance. The analogy breaks here: in a discussion people speak in turn, while attention reads every position at once.

Cross-attention in plain words

Cross-attention lets one sequence read another, which is how a translation decoder looks at the source sentence. In the same paper's encoder-decoder attention, "the queries come from the previous decoder layer" and "the memory keys and values come from the output of the encoder". The encoder is the stack that reads the input sequence, and the decoder is the stack that writes the output sequence. The formula is unchanged; only the source of K and V differs, so the score matrix is decoder length by encoder length, not square. It needs no causal mask, because the whole source is known.

The causal mask in plain words

The causal mask stops a position from reading its own future. The paper's decoder self-attention lets each position attend to all positions "up to and including that position", implemented "by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections". Without it, a model trained to produce the next token could simply copy it from the input. Encoder-only models such as Bidirectional Encoder Representations from Transformers (BERT) use no such mask, because they learn from both left and right context; the paper that introduced the generative pre-trained transformer (GPT) called GPT-3 describes an autoregressive language model, one that produces text left to right.

Text description of the diagram

• Encoder self-attention and decoder self-attention take queries, keys and values from the layer below in their own stack, and only the decoder's carries a causal mask.
• Cross-attention takes queries from the decoder and keys and values from the encoder output.

The diagram lists the three blocks. Encoder self-attention and decoder self-attention take queries, keys and values from the layer below in their own stack, and only the decoder's carries a causal mask. Cross-attention takes queries from the decoder and keys and values from the encoder output.

Worked example

Four decoder tokens, six encoder tokens, vectors of width eight, all random with a fixed seed. Every number is computed, and the data is illustrative.¹

Here is the function that does all three blocks; only the inputs and the mask change.

def attention(xq, xkv, mask=None):
    q, k, v = xq @ Wq, xkv @ Wk, xkv @ Wv
    s = q @ k.T / d**0.5
    if mask is not None:
        s = s.masked_fill(~mask, float("-inf"))
    return s.softmax(-1) @ v

causal = torch.ones(4, 4, dtype=torch.bool).tril()

In a real model each block has its own learned Wq, Wk and Wv; the saved script shares one set for illustration. The function is the one in the saved script: xq supplies the queries and xkv the keys and values. Call it with the same tensor twice for self-attention, with the decoder and encoder tensors for cross-attention, and pass causal for the decoder.

With the mask, row 0 puts all of its weight, 1.0, on token 0, and row 3 spreads it over four tokens as 0.2479, 0.2474, 0.3254 and 0.1792, which sum to one (rounded to four places).¹ The cross-attention output has the shape of the queries, four by eight, whatever the encoder length.

Text description of the diagram

• Changing token 3 changes outputs 0 to 2 by 0.0 with a causal mask and by 0.5899 without.
• Row 0 puts 0.0 weight on future tokens with a minus-infinity mask and 0.7626 with a zero mask.

Now the test that proves the mask works. Replace only the last token and rerun. With the causal mask, the outputs at positions 0 to 2 change by exactly 0.0, and only position 3 moves, by 0.3464. Without the mask, the same change moves positions 0 to 2 by up to 0.5899.¹ That is the property a language model needs: earlier outputs cannot depend on later tokens.

Finally, why minus infinity. Setting the masked scores to 0 instead leaves them in the softmax. In this example row 0 then puts 0.7626 of its weight on the three future tokens, against 0.75 if it were uniform, while the minus-infinity mask puts exactly 0.0 there, because the exponential of minus infinity is zero.¹ The PyTorch documentation for scaled_dot_product_attention (version 2.14, read Oct 2026) builds its causal option the same way: a lower-triangular mask via tril, then masked_fill_ with -inf. Our manual version matched is_causal=True to within 1e-6.¹

The reading: the causal mask is a lower-triangular filter on the scores, and it must be applied before the softmax.

Trade-offs and edge cases

The choice between the three blocks is a choice about where information lives and whether a position may see ahead.

Sources for each row: Devlin et al. (2018) for BERT's bidirectional context, Brown et al. (2020) for an autoregressive model (architecture as GPT-2), and Vaswani et al. (2017) for the encoder-decoder stack.

A fully masked row breaks. (A padding mask hides filler tokens added to make sequences the same length.) If every score in a row is minus infinity, the softmax divides zero by zero and returns not a number (NaN); the script confirms it. This appears with padding masks combined with causal masks, so keep the diagonal unmasked or handle empty rows.

The mask is not free to forget at inference. When generating, each new token is the last position, so it may attend to every cached key and value; the causal rule is then satisfied by construction, and the key-value (KV) cache stores those keys and values. The mask matters again when a prompt is processed in parallel, as in training.

A causal flag on a cached decode step attends to one key. In the PyTorch 2.14 documentation, a non-square causal mask is aligned to the upper-left corner, so is_causal=True with one new query over n cached keys lets that query attend only to the first key; the output is then that first value vector. Pass no mask for that step, as described above.

Do not set both mask options at once. The PyTorch reference asserts that attn_mask is empty when is_causal is true, and the documentation says an error is raised if both are set. Kernels (routines that run on a graphics processing unit) that fuse the mask, as in FlashAttention, avoid building the full matrix.

Text description of the diagram

• If a layer must read a separate sequence, add cross-attention.
• If the layer predicts the next token from earlier ones, add a causal mask.
• A decoder in an encoder-decoder model passes both checks, so it has a causal mask and cross-attention.

The flow is our reading of the sources above. If a layer must read a separate sequence, add cross-attention; if the layer predicts the next token from earlier ones, add a causal mask; if it reads a whole input at once, use plain self-attention.

How to answer it in an interview

The spoken structure is: one formula, the three sources, the mask before softmax, then why minus infinity.

Common mistakes

"Cross-attention is a different mechanism." (wrong) It is the same formula with K and V from another sequence. (right) "Only the source of K and V changes."

"Mask the weights after the softmax." (wrong) The remaining weights no longer sum to one. (right) "Mask the scores before the softmax so the rest renormalise."

"Use zero for the mask." (wrong) A zero score still earns weight. (right) "Use minus infinity, which gives exactly zero weight."

"Encoder self-attention needs a causal mask." (wrong) Encoder-only models such as BERT read both directions. (right) "Only autoregressive decoding needs one."

Follow-up questions

How do you implement causal masking in PyTorch?

Build a lower-triangular boolean matrix with torch.ones(T, T, dtype=torch.bool).tril(), fill the scores where it is false with minus infinity, then apply softmax, as in the code above. Or call scaled_dot_product_attention with is_causal=True, which does the same.

Where do Q, K and V each come from in decoder causal attention?

All three come from the previous decoder layer's output, through three separate learned projections. In the decoder's cross-attention block, only the queries come from the decoder; keys and values come from the encoder output.

Why use minus infinity and not zero in the mask?

Because the softmax exponentiates the scores: the exponential of minus infinity is exactly zero, while a zero score becomes a weight of one before normalising. In the worked example the zero mask left 0.7626 of row 0's weight on future tokens.

What is the difference between encoder-only, decoder-only and encoder-decoder models?

Encoder-only models such as BERT use bidirectional self-attention; decoder-only models use masked self-attention and generate left to right, as GPT-3 does; encoder-decoder models add cross-attention so the decoder reads the encoder output.

How many scores does a causal mask keep for T tokens?

T(T+1)/2 of the T² scores: each position keeps itself and the positions before it. For 2,048 tokens that is 2,098,176 of 4,194,304, or 50.02%, so the share approaches half as T grows.

Does the causal mask still matter when generating text?

It matters during training and when a prompt is processed in parallel. During one-token-at-a-time generation the new token is the last position and sees everything cached, so the rule holds without an explicit mask. The transformers hub covers the full generation loop.

Sources

1. Attention Is All You Need — arXiv (Vaswani et al.), v7 revised 2023-08-02, 12 Jun 2017
2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv (Devlin et al.), v2 revised 2019-05-24, 11 Oct 2018
3. Language Models are Few-Shot Learners — arXiv (Brown et al.), v4 revised 2020-07-22, 28 May 2020
4. torch.nn.functional.scaled_dot_product_attention — PyTorch 2.14 documentation (read 2026-10-24), 24 Oct 2026