Transformer architecture: how transformers work, from tokens to text

In this article

1. Start here: the learning path
2. How does a transformer turn text into text?
3. Tokenisation: why text becomes tokens
4. Self-attention and its variants
5. Positions, and why attention alone is not enough
6. The cost of attention and how FlashAttention changes it
7. Mixture of experts
8. Context windows
9. Encoder-only, decoder-only and encoder-decoder models
10. Questions interviewers ask on transformers
11. All 31 guides in Transformers and attention
12. Follow-up questions
13. Sources

Start here: the learning path

Transformer

A transformer is a neural-network architecture that processes a sequence of tokens in stacked layers, each combining a self-attention step with a feed-forward step. It replaced recurrence with attention as the main way tokens exchange information.

Self-attention

Self-attention is an operation in which each token builds its new representation as a weighted average of the value vectors of all tokens in the sequence, with weights from a softmax over query-key similarity scores.

Feed-forward network

A feed-forward network is the part of a transformer layer that transforms each token's vector separately, with no mixing between tokens, using an expansion and a projection around a nonlinearity, while attention handles the mixing.

Read the token journey below first, because every topic here is one stage of it. Then take the topics in this order: tokenisation, attention and its variants, positions, the cost of attention, and finally the design choices that scale models up. Each linked article has a worked example and an answer written for an interview.

This guide sits beside the deep learning hub for the network basics a transformer builds on, and the LLM inference hub for what happens when a trained model generates text.

Text description of the diagram

• The guide covers tokenisation, attention and its variants, positions, attention cost, mixture of experts and context windows.
• Read the token journey first, because every topic here is one stage of it.

1. What is byte pair encoding? How BPE tokenization works
2. What is self-attention, and how does it work step by step?
3. What is a mixture of experts model, and how does it work?
4. What is multi-head attention in a transformer?
5. What is rotary positional embedding (RoPE)?
6. Why is attention quadratic, and how do you make it cheaper?
7. What is FlashAttention?
8. What is a context window in an LLM, and what does it cost?
9. Why are attention scores divided by sqrt(d_k) in a transformer?
10. How encoder-only, decoder-only and encoder-decoder transformers read and write
11. How does Sora work, step by step?
12. What are LLM scaling laws, and what does Chinchilla say?
13. Why is standard attention memory-bound, and how does FlashAttention fix it?
14. What is the difference between transformers and RNN?
15. What are the differences between Transformer and LLaMA's layer normalization?
16. Explain from a high level point of view how a new token is generated?
17. In terms of token size, how long should each token be?
18. What breaks first when you push context length way up?
19. How do you evaluate long-context models (needle-in-a-haystack and beyond)?
20. Why does attention require scaling?

How does a transformer turn text into text?

A decoder-only transformer turns a prompt into one next token by running the tokens through the same stack of layers and reading a probability for each vocabulary entry from the last position. The loop then appends the chosen token and repeats. Six stages cover it.

Tokenise. A token is a word or word fragment that the model reads as one unit. The tokeniser splits the text into tokens and maps each to an integer ID. Byte pair encoding is a widely used scheme for building that vocabulary; the byte pair encoding page shows how.

Embed and add position. Each ID selects a learned vector, and a positional encoding is added or mixed in so word order is not lost.

Attend. For every token the model forms a query, a key and a value vector. PyTorch documents the operation as softmax of the query-key product times a scale, plus a bias, applied to the values, with the scale defaulting to one over the square root of the key (per-head query) size, d_k. In a decoder the causal option puts minus infinity above the diagonal so a token cannot read later tokens. In the original design each of the two sub-layers, attention and feed-forward, is wrapped in a residual connection followed by layer normalisation (Vaswani et al., 2017, §3.1).

Transform. A feed-forward network is applied to each token separately, with no mixing between tokens. Attention mixes information across tokens; the feed-forward network processes each token.

Repeat and read out. The attend-and-transform block repeats in every layer, and the last layer's output at the final position is projected to scores. A logit is one raw score per vocabulary entry. The cross-entropy loss trains those scores against the true next token.

Sample. A softmax turns the logits into probabilities and the model picks the next token. Temperature rescales the logits first.

A three-token example you can check by hand

Take the token vectors (1, 0), (0, 1) and (1, 1), use each vector as its own query and key, and divide each dot product by √2, the square root of the key size. With these illustrative vectors, token 2 scores 0 and 0.707, and token 3 scores 0.707, 0.707 and 1.414. Each row of weights is a softmax over the scores the causal mask allows:

Causal attention gives the first token weight 1.0 on itself. The second token weights the first at 0.330 and itself at 0.670, and the third weights the first two at 0.248 each and itself at 0.503. Each row adds to 1, and no token has weight on a later one¹.

For sampling, take illustrative logits of 3.0, 2.0, 0.5 and −1.0, and divide each by the temperature T before the softmax:

At temperature 1.0 the probabilities are 0.681, 0.251, 0.056 and 0.012. At temperature 0.5 the distribution sharpens to 0.875, 0.118, 0.006 and 0.0003¹.

Text description of the diagram

• Attention mixes information across tokens and the feed-forward network processes each token separately.
• The last layer's output at the final position is projected to one logit per vocabulary entry.
• A softmax with temperature turns the logits into probabilities for the next token.

These are the sizes of the base model in Vaswani et al., 2017, §3.3 (d_model = 512, d_ff = 2,048); counts ignore biases and normalisation parameters¹. The pattern, with the feed-forward network holding about two thirds of a block's weights when its width is four times the model width, is worth remembering.

Tokenisation: why text becomes tokens

Tokenisation exists because a model needs a fixed vocabulary and a word list would be unbounded. Byte pair encoding starts from small units and repeatedly merges the most frequent adjacent pair, so common words become one token and rare words split into pieces. The vocabulary size is a trade-off between sequence length and the size of the embedding table: a bigger vocabulary means fewer tokens per sentence but more rows to store and learn, and rare tokens that get few training examples. Read byte pair encoding for the step-by-step merge and the interview questions about vocabulary size.

Self-attention and its variants

Self-attention lets every token read every other token in one step, so distant words are connected directly instead of through a chain of steps. Variants change which tokens may be read. Self-attention reads the same sequence, cross-attention reads a second sequence such as an encoder's output, and masked attention hides later tokens during generation. Multi-head attention runs several attention operations with separate learned projections and concatenates them, and grouped-query and multi-query variants share keys and values across heads to shrink the cache.

Positions, and why attention alone is not enough

Attention treats its input as an unordered set, so a model with no position information cannot tell "dog bites man" from "man bites dog". The original design added sinusoidal encodings; many current models rotate the query and key vectors by position-dependent angles, an approach called rotary positional embedding. The choice affects how well a model handles sequences longer than it was trained on. Rotary encodings act on the query and key vectors only, while the original additive encodings change the embeddings themselves.

The cost of attention and how FlashAttention changes it

Attention compares every token with every other, so the number of scores per head grows with the square of the sequence length. Going from 2,048 to 8,192 tokens multiplies the tokens by 4 and the scores by 16, from 4,194,304 to 67,108,864 per head per layer¹. The FlashAttention article shows how tiling avoids storing that matrix without changing the answer, and the KV cache article covers the memory cost at inference. Numerical formats matter here too: see fp16 versus bf16 and the graphics processing unit (GPU) fundamentals page.

Text description of the diagram

• Going from 2,048 to 8,192 tokens multiplies the tokens by 4 and the scores by 16.
• At 8,192 tokens there are 67,108,864 scores per head per layer.

Mixture of experts

A mixture-of-experts layer replaces one feed-forward network with several, and a small router sends each token to a few of them. The model has many more parameters than it uses on any one token, which raises capacity without a matching rise in per-token compute. The costs are load balancing across experts and communication between devices.

Context windows

The context window is the maximum number of tokens a model can attend to in one pass. It is set by training, by the position scheme and by memory: attention cost and cache size both grow with length. A longer window helps only if the model uses distant tokens well, which is a separate question from whether it fits.

Encoder-only, decoder-only and encoder-decoder models

The three families differ in what each token may see. Bidirectional Encoder Representations from Transformers (BERT) is an encoder pre-trained on unlabelled text using both left and right context in every layer, then adapted with fine-tuning through one added output layer; its authors report an 80.5% score on the General Language Understanding Evaluation (GLUE) benchmark. A decoder-only model such as the 175-billion-parameter model in Language Models are Few-Shot Learners is autoregressive and is steered by text prompts and examples rather than gradient updates. The original transformer paper was an encoder-decoder for translation and reported 28.4 bilingual evaluation understudy (BLEU) on English to German and 41.8 on English to French for a single model.

Questions interviewers ask on transformers

Transformer questions in LLM-depth interview rounds come in a small set of shapes: explain attention from first principles, walk through how a new token is generated, say why attention cost is quadratic, and compare model families. Dedicated lists are in LLM interview questions and the natural language processing (NLP) interview questions; the sections above and the follow-ups below cover the same ground, and the AI agents hub shows where these models are used.

Follow-up questions

What is transformer architecture in AI?

It is a neural-network design that uses self-attention instead of recurrence to relate every token in a sequence to every other token, in stacked layers of attention and feed-forward blocks. The deep learning hub covers the network basics it builds on.

Who invented transformer architecture?

It was introduced in the 2017 paper Attention Is All You Need by Vaswani and seven co-authors, which showed an attention-only translation model reaching 28.4 BLEU on English to German.

What is transformer architecture best known for?

Large language models: BERT-style encoders for understanding tasks and generative pre-trained transformer (GPT)-style decoders for generation. The GPT-3 paper describes a 175-billion-parameter autoregressive model that performs tasks from text prompts alone.

Why does attention cost grow with the square of the sequence length?

Every token is scored against every other, so $n$ tokens make $n^2$ scores per head. The FlashAttention article shows why storing them is the bottleneck.

What does the model output at each step of generation?

A raw score, called a logit, for every vocabulary entry, which a softmax turns into a probability distribution from which the next token is chosen. The LLM inference hub explains how the loop is made fast with a cache.

Sources

1. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) — arXiv (Devlin et al.), 11 Oct 2018
3. Language Models are Few-Shot Learners (arXiv 2005.14165) — arXiv (Brown et al.), 28 May 2020
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135) — arXiv (Dao et al.), 27 May 2022
5. torch.nn.functional.scaled_dot_product_attention, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026