In this article
Before you start
In retrieval-augmented generation (RAG), Transformers play two roles: one turns text into embeddings for search, and one writes the answer. This page builds on embeddings and vector search.
Why interviewers ask this
Almost every modern language and retrieval system sits on a Transformer, so a wrong picture of it makes every later answer shaky. The question separates a candidate who has memorised a name from one who can say what the model computes and what that costs.
A weak answer says "it is the architecture behind ChatGPT." A strong answer says it is built on attention, names the encoder, decoder and image families, and gives one use of each. A Staff-level answer adds the trade-off: every position scores every other position, so cost rises with the square of the input length, and that limit shapes context windows, chunk sizes and latency in a RAG system. A good answer also corrects the common confusion that "Transformer" and "large language model" are the same thing.
Transformer from first principles
- Dot product
- A dot product is an operation that multiplies two equally long lists of numbers position by position and adds the results, giving one score that is larger when the lists hold larger numbers and point in similar directions, which attention uses to compare tokens.
In plain words
Picture a meeting where every person may glance at every other person's notes before writing their own summary line. How much each person copies from each set of notes depends on how relevant it looks. A Transformer does this with numbers, many times in a row. The analogy breaks in one place: in a real meeting people speak in turn, but a Transformer reads all the notes in one pass.
In one precise sentence, the original paper proposes a network based solely on attention, "dispensing with recurrence and convolutions entirely". The authors introduced it for machine translation. Attention itself was not new: the same abstract says the best earlier models already linked encoder and decoder through attention. What was new was dropping everything else.
Step 1: Text becomes vectors with position information
The model first splits the input into tokens, then looks each one up in a table to get an embedding, a list of numbers. Self-attention is permutation-equivariant: reordering the tokens only reorders the outputs, so it cannot tell order by itself. So the model must be told where each token sits, and the original design adds position information to the embeddings before the first layer.
Input: raw text or image patches. Output: one vector per token, carrying both meaning and position. A reader now knows what goes into the attention step: a stack of n vectors, one per token. Everything after this works on that stack, and none of it needs to wait for the token before it, which is why training can use many tokens at the same time.
Step 2: Every token scores every other token, so cost grows with the square
Each token vector yields a query (what the token looks for), a key (what it offers) and a value (what it passes on), using learned weights. The score of token i against token j is the dot product of query i and key j, divided by the square root of the key size to keep scores moderate.
Each of n tokens scores all n tokens, so one head in one layer builds n times n scores. An illustrative 224-pixel image cut into 16-pixel patches gives 14 by 14, or 196 patches, so 38,416 scores (a real Vision Transformer adds a class token). For 512 tokens it is 262,144 scores and for 2,048 tokens it is 4,194,304, which is 16 times more for 4 times the length.1
| Quantity | Value (computed, illustrative lengths) |
|---|---|
| Scores for 196 image patches | 38,416 |
| Scores for 512 tokens | 262,144 |
| Scores for 2,048 tokens | 4,194,304 |
| Growth from 512 to 2,048 tokens | 16 times |
| Storage if the 2,048-token table were stored in full, in 16-bit numbers | 8,388,608 bytes |
Text description of the diagram
- The score table has 38,416 entries for 196 patches, 262,144 for 512 tokens and 4,194,304 for 2,048 tokens.
- Quadrupling the length makes the table 16 times larger.
The 196-patch bar is short, so read the printed values. Compute stays quadratic per head and layer, but fused kernels such as FlashAttention (Dao et al., 2022) work in tiles without storing the full table, so memory grows linearly. The square still makes long inputs expensive; see where RAG latency and cost go.
Step 3: Softmax turns scores into weights that mix the values
Softmax converts each token's row of scores into positive weights that add to one. The token's new vector is the weighted sum of all the value vectors. A token that matches strongly pulls in more of that token's value.
A tiny illustrative case with a key size of 4 shows it. Token 1 has query (1, 0, 1, 0) and the three keys are (1, 0, 1, 0), (0, 1, 0, 1) and (1, 0, 0, 1). The dot products are 2, 0 and 1; divided by the square root of 4, which is 2, they become 1, 0 and 0.5. Softmax gives weights of 0.506, 0.186 and 0.307. With values (1, 0), (0, 1) and (1, 1), the mixed output for token 1 is (0.814, 0.494).1
Input: scores. Output: one new vector per token that now contains information from the whole sequence. Real models run many such heads side by side, each with its own learned weights, and join their outputs.
Step 4: Blocks stack, and three families use them differently
A block is the attention step followed by a small feed-forward network, each sub-layer wrapped in a residual connection (its input is added back to its output) and layer normalisation, and blocks are stacked. The families differ in which tokens each token may see, meaning which positions it may attend to.
An encoder-only model such as BERT (Bidirectional Encoder Representations from Transformers) conditions on both left and right context and, after pre-training, needs just one extra output layer to reach state-of-the-art results on eleven language tasks. A decoder-only model of the Generative Pre-trained Transformer (GPT) family uses a left-to-right design where each token attends only to earlier tokens, which suits writing text one token at a time.
The original paper used both an encoder and a decoder for translation, and its abstract reports results on two machine translation tasks. The Vision Transformer (ViT) feeds a sequence of image patches to a standard Transformer. Beyond text and images, Whisper uses a standard encoder-decoder Transformer for speech recognition (see its Section 2.2), and Sentence-BERT derives sentence embeddings for semantic similarity search.
Step-by-step description
- Tokens become vectors with position.
- Every token scores every token.
- Softmax weights mix the values.
- Blocks are stacked.
- Tokens become vectors with position information.
- Every token scores every other token, so the score table has n by n entries.
- Softmax weights mix the value vectors.
- Blocks are stacked and the families differ in which tokens each token may see.
Text description of the diagram
- BERT lets each token see left and right context.
- GPT-style models let each token see only earlier tokens.
- The Vision Transformer feeds image patches to a standard Transformer.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What is a Transformer, and where is it used?
A Transformer is a neural network that mixes tokens only through attention. Each token makes a query, a key and a value, scores every other token, and takes a weighted mix of their values, so it needs no recurrence and trains in parallel. Encoder-only models like BERT read both directions for understanding, decoder-only models generate left to right, and Vision Transformers apply the same blocks to image patches. In RAG it embeds chunks for search and writes the final answer. The cost is the catch: the score computation grows with the square of the input length, which limits context and drives latency.
What makes it Staff-level is the last sentence: it names the quadratic cost and ties it to a system consequence instead of stopping at the definition.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "The model behind ChatGPT." | Network that mixes tokens only through attention. | Also says attention predates it; dropping recurrence was the change. |
| Depth | Names no mechanism. | Query, key, value, softmax, weighted sum. | Adds position information and why it is needed. |
| Breadth of use | Text only. | Encoder, decoder and image families. | Maps each family to a task, including RAG roles. |
| Trade-offs | None. | Mentions long inputs are costly. | States the square, with a number, and its effect on context and latency. |
| Communication | Jargon first. | Definition, then example. | Definition, example, trade-off in about a minute. |
Three variants of this question
How would you explain a Transformer to someone non-technical?
Say it is a program that, for every word, looks at all the other words and decides how much each matters before it writes its own summary. Then give one use, such as search or translation.
Is a Transformer only for text?
No. The Vision Transformer applies the same architecture to image patches, and the paper reports that a pure Transformer on patches can perform very well on image classification. The architecture was first used for machine translation (Vaswani et al., 2017), and it also serves speech recognition (Radford et al., 2022) and search embeddings (Reimers and Gurevych, 2019).
What if the input is four times longer?
Each head scores every pair of tokens, so the score table is 16 times larger, as in the worked numbers above. That is one reason long contexts are expensive and encoders cap input length, which chunking strategies for RAG respect.
Common mistakes
- "A Transformer is a large language model." (wrong) A Transformer is an architecture; a language model is one thing built with it, and BERT and ViT are not text generators. (right) "A language model is one use of the Transformer architecture."
- "The Transformer paper invented attention." (wrong) The abstract says earlier models already connected encoder and decoder through attention. (right) "It removed recurrence and convolution and kept attention."
- "BERT and GPT work the same way." (wrong) BERT reads both directions; GPT-style models read left to right. (right) "BERT is an encoder for understanding; GPT-style models generate."
- "Attention cost is linear in length." (wrong) Every token scores every token. (right) "The score table grows with the square of the length."
Follow-up questions
Is a Transformer the same as a large language model?
No. A large language model is a very large model trained on text, and most are built from Transformer blocks, but the Transformer also underlies encoders for search and image models. The architecture is the blueprint; the model is one building made from it.
What is the difference between BERT and GPT?
BERT is an encoder pre-trained by masking some input tokens and recovering them, so it sees both sides of a token. GPT-style models attend only to earlier tokens, which suits generation. Both use Transformer blocks.
Why does a Transformer need position information?
Because self-attention is permutation-equivariant (reordering the tokens only reorders the outputs), the model would not know "dog bites man" from "man bites dog". Position information added to the embeddings restores order.
How are Transformers used in RAG?
An encoder Transformer turns each chunk and the question into embeddings for vector search, and a decoder Transformer writes the answer from the retrieved text. See the RAG guide for the full pipeline, how to evaluate a RAG system and common RAG interview questions.
Did the Transformer paper invent attention?
No. Its abstract notes that the best earlier translation models already used attention between encoder and decoder. The paper's claim is a network based solely on attention.
- A Transformer builds each token's representation from attention over all tokens, with no recurrence or convolution.
- Each token makes a query, key and value; softmax weights then mix the values.
- Encoder-only, decoder-only and image Transformers differ in which positions may attend to which.
- The score table grows with the square of the input length: 16 times larger for 4 times the length.
- Say the definition, one use per family, then the cost.
Sources
- Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
- Attention Is All You Need (NeurIPS 2017 full text) — NeurIPS (Vaswani et al.), 4 Dec 2017
- Robust Speech Recognition via Large-Scale Weak Supervision (arXiv 2212.04356) — arXiv (Radford et al.), 6 Dec 2022
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv 1908.10084) — arXiv (Reimers and Gurevych), 27 Aug 2019
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135) — arXiv (Dao et al.), 27 May 2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) — arXiv (Devlin et al.), 11 Oct 2018
- BERT, full text with the comparison to left-to-right models (ar5iv copy) — arXiv / ar5iv (Devlin et al.), 11 Oct 2018
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv 2010.11929) — arXiv (Dosovitskiy et al.), 22 Oct 2020
