How encoder-only, decoder-only and encoder-decoder transformers read and write

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the transformer architecture learning path.

Why interviewers ask this

Candidates report this question, in several wordings, in interviews at large technology companies and AI companies. It checks whether you know that "transformer" is a family and not one model. Bidirectional Encoder Representations from Transformers (BERT) and the generative pre-trained transformer (GPT) models, whose name is covered in what GPT stands for, are both transformers.

A weak answer says "BERT is for understanding and GPT is for generation" and stops. A strong answer says why: the models differ in the attention mask, the rule that decides which tokens may look at which, and that rule decides the training task, the cost and the natural use. A Staff-level answer adds that the clean three-way split is blurrier in practice, because prefix variants sit between the boxes, and that the best choice depends on the training objective and on later tuning.

What each one is

Attention mask

An attention mask is a grid of ones and zeros, one row and one column per token, that tells self-attention which tokens each token is allowed to look at; a zero blocks that pair, so the mask decides what information can flow.

Autoregressive

Autoregressive is a property of a sequence model that generates one token at a time, feeding each output back in as input, so every new token depends on the tokens before it and never on later ones, as in decoder-only text generation.

In plain words

Picture three office workers. An editor reads a whole document, front to back and back to front, before marking it up. A novelist writes one word at a time and cannot know what comes next. A translator first reads the whole source text, then writes the translation while glancing back at the source. The three map to an encoder-only, a decoder-only and an encoder-decoder model. The analogy breaks in one place: real models do not read or write in sequence inside a layer; the mask only controls what each token may see.

In one precise sentence, the three designs use the same attention blocks and differ in the mask and in whether a second stack is attached.

Step 1: The attention mask decides who sees whom

The attention mask is the single switch behind the whole comparison. In self-attention every token scores every other token; the mask then blocks the pairs that are not allowed. In practice the blocked scores are set to minus infinity before the softmax, so their attention weights become exactly zero. Multiplying the scores by zero would not work, because exp(0) is one and the blocked pairs would still get weight. A full mask of all ones lets every token see every token. A causal mask keeps only the lower triangle.

For five tokens, a full mask allows 25 pairs and a causal mask allows 15.¹ Because no token can see a later one under the causal mask, a decoder can be trained on a whole sentence at once while still learning to output each next token. You now know the one idea the next three steps reuse.

Step 2: Encoder-only models read in both directions

An encoder-only model uses the full mask. The original Transformer of Vaswani et al. (2017) had an encoder and a decoder, and an encoder-only model keeps just the encoder half. The name of BERT (Devlin et al., 2018) says it: Bidirectional Encoder Representations from Transformers. The paper describes pre-training that conditions on both left and right context in all layers.

Masked language modeling (MLM) is the training task that makes this work: some input tokens are hidden and the model must recover them from the tokens on both sides, as Devlin et al. (2018), §3.1 describe. The same section describes a second pretraining task, next sentence prediction (NSP), in which the model predicts whether one sentence follows another in the original text. Devlin et al. report that one extra output layer is enough to tune BERT for tasks such as question answering and language inference. The model produces representations, not new text, so it suits classification, tagging and search.

Step 3: Decoder-only models write left to right

A decoder-only model uses the causal mask. It is trained to output the next token given all earlier ones; a model that generates a sequence one token at a time, feeding each output back in as input, is called autoregressive. Hao et al. (2022), the MetaLM paper, note that GPT-3 showed in-context abilities emerge from causal language model pretraining.

The same paper says "causal language models are good at zero-shot or in-context learning, while non-causal models perform better for finetuning". One consequence follows from the mask itself: adding a new token never changes what earlier tokens saw, so their stored keys and values can be reused. That reuse is the idea behind the key-value (KV) cache used when serving these models. The model needs no separate reader stack, which is one reason the design is simple to build.

Step 4: Encoder-decoder models read first, then write

An encoder-decoder model has two stacks. The encoder reads the source text with a full mask. The decoder writes the output with a causal mask and, in each layer, uses cross-attention to look at the encoder's output. Vaswani et al. (2017) built the first Transformer this way for translation and connected the two stacks through attention. The older network family it descends from is covered in seq2seq models.

Raffel et al. (2019) introduced T5 (Text-to-Text Transfer Transformer), an encoder-decoder model that casts every language problem as text-to-text, so summarization, question answering, classification and translation share one format. The price is two stacks and the extra cross-attention step.

1 · The mask2 · Encoder-only3 · Decoder-only4 · Encoder-decoderReplay

Step-by-step description

1. The mask decides who sees whom.
2. Every token sees both directions.
3. Each token sees earlier tokens only.
4. Read with one stack, write with another.

• The attention mask decides which tokens each token may see.
• An encoder-only model lets every token see both directions.
• A decoder-only model lets each token see only earlier tokens.
• An encoder-decoder model reads with an encoder and writes with a decoder.

How do the three designs differ side by side?

The mask and the second stack decide the job each design does best, and the table puts the three next to each other.

Worked example

Counting allowed attention pairs shows what each mask removes. The sizes below are illustrative, and the counts are arithmetic.¹ For five tokens, the full mask allows 25 pairs, the causal mask 15, and a prefix mask with a two-token prefix 16, because the two prefix tokens see each other and every later token sees everything before it.¹ The causal mask for five tokens is:

1 0 0 0 0
1 1 0 0 0
1 1 1 0 0
1 1 1 1 0
1 1 1 1 1

Now add a task: translate a five-token source into a three-token target. An encoder-decoder model allows 25 pairs in the encoder, 6 in the decoder and 15 in cross-attention, 46 in total.¹ A decoder-only model reading the same eight tokens with a causal mask allows 36 pairs. A prefix model with the five source tokens as its prefix allows 46, the same count, but with one set of weights.¹

Each count is per attention head in one layer, which is why the cost of long inputs is a topic of its own: see why attention is quadratic and how to make it cheaper. Each head in multi-head attention applies the same mask.

Trade-offs and edge cases

No design wins everywhere. Wang et al. (2022) note that architectures and pretraining objectives differ widely across state-of-the-art models and that systematic comparisons were limited. They compared causal and non-causal decoder-only models and encoder-decoder models, each trained with autoregressive or masked language modeling objectives, with and without multitask prompted finetuning. They trained 5-billion-parameter models on 170 billion tokens.

Causal decoder-only models trained autoregressively showed the strongest zero-shot generalization after unsupervised pretraining only, but models with non-causal visibility of the input, trained with masked language modeling and then multitask finetuning, performed best overall. The Staff-level point is their adaptation result: a pretrained causal decoder can be efficiently adapted into a non-causal one, so the boxes are less rigid than the names suggest.

Edge cases an interviewer may probe:

• Prefix models blur the boxes: the input gets a full view, the output a causal view, and there is only one stack.
• Encoder-only models are not built to generate: the full mask gives no left-to-right order to follow, although masked tokens can be filled in repeatedly.
• Encoder-decoder is not obsolete. T5 remains a reference text-to-text design (Raffel et al., 2019), and its two stacks suit tasks whose input and output differ.
• Cost differs. Cross-attention adds work per decoder layer, while a decoder-only model has only the causal mask to manage. Raffel et al. (2019, §3.2) compare the two: an encoder-decoder with L layers in each stack has about twice the parameters of a decoder-only model with L layers, at a similar compute cost.

The state of practice as of October 2026 is that all three layouts are in current use. Decoder-only is the layout most open large language models use, for example Llama 2 (Touvron et al., 2023). Encoder-only models remain current for embeddings, retrieval and classification, for example ModernBERT (Warner et al., 2024). Encoder-decoder remains current where the input is a separate signal, for example Whisper speech recognition (Radford et al., 2022). One serving point follows: in an encoder-decoder model the encoder output is computed once per input and reused at every decoding step, so only the decoder grows a KV cache as it writes.

The decision flow shows the natural fit, not a requirement. If the output maps a separate, fully known source, such as a translation, a summary or a transcript, an encoder-decoder model fits. In 2025–26 practice decoder-only models also translate and summarize from a prompt.

Text description of the diagram

• If the model need not write text, an encoder-only model fits.
• If the output maps a separate, fully known source, an encoder-decoder model fits.
• Decoder-only models use a causal mask and output the next token.

How to answer it in an interview

The spoken structure: name the mask, give one model and one job per design, then state the trade-off. A rehearsal out loud takes about a minute.

Common mistakes

• "The Transformer means encoder-decoder, so GPT is not a transformer." (wrong) The original design had both stacks, but decoder-only and encoder-only models reuse its parts. (right) "All three are transformers; they keep different stacks."
• "BERT generates text like GPT." (wrong) BERT is a representation model tuned with an added output layer. (right) "BERT produces representations; GPT writes the next token."
• "Bidirectional means the model reads the text twice." (wrong) BERT conditions jointly on left and right context in every layer. (right) "A full mask lets each token see both sides in one pass."
• "Decoder-only always beats encoder-decoder." (wrong) Results depend on the objective and on tuning. (right) "It depends on the pretraining objective and on multitask finetuning."

Follow-up questions

Is encoder-decoder or decoder-only better?

Neither wins everywhere. Decoder-only is the simplest layout, and in the experiments of Wang et al. (2022) causal decoder-only models showed the strongest zero-shot generalization after unsupervised pretraining only; models with non-causal visibility of the input, trained with masked language modeling and then multitask finetuning, performed best overall.

What is the difference between an encoder-only architecture and a decoder-only architecture?

An encoder-only model uses a full mask and outputs representations; a decoder-only model uses a causal mask and outputs the next token. So the first fits understanding tasks and the second fits generation.

What are the types of LLM models in the market, and how do they differ?

By layout there are the three types on this page, plus prefix variants between them. They differ in mask, training task and natural use. Open models with published layouts include BERT and RoBERTa (encoder-only), Llama (decoder-only) and T5 and BART (encoder-decoder). The large language model (LLM) interview questions page covers neighbouring questions; avoid claiming the layout of a named proprietary model unless its maker publishes it.

Why can BERT not generate text the way GPT does?

BERT is not built to generate: its full mask lets every token see later tokens, so there is no left-to-right order to continue, and its training task fills in hidden tokens instead of outputting the next one. Filling masks repeatedly is possible but is not a natural writer.

How would you count the parameters of an encoder or a decoder?

Add the embedding table, then for each layer the attention weights and the feed-forward weights, and multiply by the number of layers. With hidden size d and a feed-forward width of 4d, a layer has about 4d² attention weights and 8d² feed-forward weights, so roughly 12·d². A decoder in an encoder-decoder model adds the weights of cross-attention. For BERT-base, Devlin et al. (2018) report 12 layers, hidden size 768 and 110 million parameters; the rule gives 84,934,656 for the layers and 23,040,000 for a 30,000-token vocabulary, 107,974,656 in all, close to the reported figure.¹ The gap is the terms the rule leaves out, such as position embeddings, biases and the output layers.

Sources

1. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) — arXiv (Devlin et al.), 11 Oct 2018
3. Llama 2: Open Foundation and Chat Models (arXiv 2307.09288) — arXiv (Touvron et al.), 18 Jul 2023
4. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (ModernBERT, arXiv 2412.13663) — arXiv (Warner et al.), 18 Dec 2024
5. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, arXiv 2212.04356) — arXiv (Radford et al.), 6 Dec 2022
6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (arXiv 1910.10683) — arXiv (Raffel et al.), 23 Oct 2019
7. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (arXiv 2204.05832) — arXiv (Wang et al.), 12 Apr 2022
8. Language Models are General-Purpose Interfaces (arXiv 2206.06336) — arXiv (Hao et al.), 13 Jun 2022