In this article
Before you start
Part of the transformer architecture path.
Why interviewers ask this
A candidate who cannot state this difference cannot explain why modern language models are built the way they are.
The question separates three levels. A weak answer says "transformers use attention and RNNs use loops." A strong answer ties the difference to parallel training and to how far a signal must travel between two tokens. A Staff-level answer also gives the cost: attention is not free, and the comparison flips with sequence length.
Transformers and RNNs from first principles
- Sequential operations
- Sequential operations are the steps in a layer that must run one after another because each needs the result of the one before. The fewer a layer needs, the more of its work a graphics card can do together.
- Maximum path length
- Maximum path length is the number of connections a signal must pass through to travel between the two most distant positions in a sequence. A short path makes it easier for a network to link far-apart tokens.
In plain words
Picture two readers given one long letter. The first reads one word at a time and rewrites a short note in their head after each word. The second spreads every word on a table and looks at any pair directly. The first is an RNN; the second is a transformer. The analogy breaks in one place: the second reader still pays for looking, because the number of pairs grows quickly with the letter's length.
A recurrent neural network is a network that reads a sequence one token at a time and passes a hidden state, its running memory, from each step to the next. In one precise sentence, an RNN has a chain of dependent steps, while a transformer layer compares all token pairs with no such chain.
Step 1: An RNN reads one token at a time
An RNN takes token t and the memory from token t minus one, and outputs a new memory. Step t cannot start until step t minus one finishes. That chain is the defining property.
Two consequences follow. A sequence of 1,000 tokens needs about 1,000 dependent steps, and a signal from the first token reaches the last through about 999 of them.1 Vaswani et al., Table 1 lists both as growing in proportion to n for a recurrent layer. Training also gets harder as the gap grows: Bengio, Simard and Frasconi (1994) show that gradient-based learning faces an increasingly difficult problem as the duration of the dependencies grows.
Step 2: A transformer links every pair of tokens directly
A transformer is a network built on attention. Vaswani et al. (2017) describe it as "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." In each layer, every token computes a weight for every other token and takes a weighted mix, using matrix operations over the whole sequence, so no step waits for another.
Per Table 1, a self-attention layer needs a constant number of sequential operations, and any two tokens are one connection apart. Training therefore runs in parallel across positions. The base model beat every earlier published result, ensembles included, after 100,000 steps, about 12 hours on eight NVIDIA P100 GPUs (Vaswani et al., Section 5.2). The larger "big" model reached 28.4 bilingual evaluation understudy (BLEU) (bilingual evaluation understudy, a translation score) on English-to-German, over two points above the previous best.
One thing is lost: without recurrence a transformer has no sense of order. Positional encoding is a set of extra numbers added to each token that tell the network where the token sits (Section 3.5). Attention is not always cheaper per layer; Step 3 shows where it flips.
Step 3: Attention costs more per layer once the sequence is long
Parallel does not mean cheap. Table 1 gives the work per layer: about n² · d for self-attention and n · d² for a recurrent layer, where n is the sequence length and d is the width of each token's vector. The paper's base model uses d = 512. Self-attention costs about n squared times d per layer, a recurrent layer about n times d squared. At 2,000 tokens self-attention costs nearly four times (3.9×) a recurrent layer.1 The two costs are equal at n = d, and the paper notes self-attention is faster when n is smaller than d.
| Sequence length n | Self-attention n²·d (millions) | Recurrent n·d² (millions) | Cheaper per layer |
|---|---|---|---|
| 100 | 5.12 | 26.21 | self-attention |
| 512 | 134.22 | 134.22 | equal |
| 1,000 | 512 | 262.14 | recurrent |
| 2,000 | 2,048 | 524.29 | recurrent |
Text description of the diagram
- Self-attention is cheaper per layer than a recurrent layer at 100 tokens and equal at 512.
- At 2,000 tokens self-attention costs nearly four times a recurrent layer.
These are operation counts with constants ignored, not timings. The transformer still wins on wall-clock training because its steps run in parallel.
Parallel training is not parallel generation. At inference a decoder-only transformer still produces one token at a time and keeps keys and values for every past token, so its memory grows with the context, while an RNN carries a fixed-size state. Recurrent-style designs such as state-space models (for example Mamba) are being revived for long contexts for this reason. The quadratic growth is why long contexts are costly; see why attention is quadratic and how to make it cheaper.
Step-by-step description
- An RNN waits step by step; attention runs together.
- Any two tokens are one connection apart in attention.
- Attention cost grows with the square of the length.
- A recurrent layer needs n sequential steps and self-attention needs one.
- The longest path between two tokens is n steps for a recurrent layer and one for self-attention.
- Self-attention costs about n squared times d per layer, a recurrent layer about n times d squared.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What is the difference between transformers and RNN?
An RNN reads tokens one at a time and carries a hidden state, so each step waits for the one before. That makes training sequential, and a signal between distant tokens must cross every step in between. A transformer uses self-attention, so every token connects directly to every other token in one layer. Training is parallel across positions and any two tokens are one connection apart.
The price is cost: attention grows with the square of the sequence length, so it is cheaper per layer only when the sequence is shorter than the vector width. Attention also ignores order, so a transformer needs positional encodings. At inference a decoder still generates one token at a time and keeps keys and values for every past token, while an RNN carries a fixed-size state.
What makes this Staff-level is the last three sentences: it states the cost that flips with length, the position fix and the inference caveat, instead of stopping at "parallel and better at long range." Say it in the order definition, parallelism, path length, cost, position.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Transformers use attention, RNNs use loops. | RNNs process tokens in sequence; transformers attend to all tokens. | Names sequential operations and path length as the measurable differences. |
| Depth | Says transformers are better. | Explains parallel training and direct token links. | Cites per-layer cost for both and where they cross. |
| Trade-offs | None. | Mentions attention memory growing with length. | States attention is cheaper only when n is below d, and why that still favors training speed. |
| Communication | Lists facts in no order. | Definition, then why it matters. | Definition, mechanism, cost, then the fix for lost order. |
| Follow-up readiness | Surprised by "is it always cheaper?" | Can explain quadratic cost. | Can discuss long-context fixes and the gradient problem in RNNs. |
Three variants of this question
Why can transformers train faster than RNNs?
A transformer layer has no chain of dependent steps, so all positions are processed together on a GPU. An RNN must finish step t minus one before step t, which limits how much work the GPU can run at once.
Are transformers always better than RNNs?
No. Per layer, attention costs more than a recurrent layer once the sequence is longer than the vector width, and its cost grows with the square of the length. Transformers win on parallel training and long-range links, not on every cost line.
What if the sequence is much longer than the vector width?
Then the n² · d term dominates and the attention layer becomes the expensive part. That is the motivation for cheaper attention variants and for managing the context window.
Common mistakes
- "Transformers are always cheaper than RNNs." (wrong) Attention is about n² · d per layer against n · d² for recurrence, so it is cheaper only when n is below d. (right) "Transformers are parallel and have short paths, but attention cost grows quadratically with length."
- "Transformers do not need position information." (wrong) Without recurrence there is no built-in order. (right) "A transformer injects positional encodings so tokens know where they sit."
- "RNNs cannot learn long dependencies at all." (wrong) They can, but gradient learning gets harder as the dependency span grows. (right) "Long spans are hard to learn with gradient descent, and the difficulty grows with the span."
- "Attention replaced the encoder-decoder idea." (wrong) The original paper keeps the encoder-decoder layout and removes recurrence and convolution. (right) "The transformer is an encoder-decoder built from attention instead of recurrence."
Follow-up questions
Do transformers need positional information?
Yes. A transformer has no recurrence, so it must inject information about the position of tokens, as Section 3.5 of the original paper states. Later designs such as rotary positional embedding change how that information is added.
Can RNNs learn long-range dependencies at all?
RNNs can learn long-range dependencies, but learning gets harder as the dependency span grows, which is the result of Bengio, Simard and Frasconi (1994). Gated variants such as long short-term memory (LSTM) networks exist to ease this. In a transformer versus LSTM comparison, an LSTM is still recurrent, so it still runs one step after another and keeps a path of about n steps between distant tokens.
What was the original transformer compared against?
Recurrent and convolutional encoder-decoder models, which were then the dominant approach. The paper's abstract reports better quality with more parallelization and significantly less training time. For the encoder-decoder idea itself, see seq2seq models.
Why is attention quadratic in sequence length?
Every token is scored against every other token, so n tokens produce n × n scores. Each score is a dot product of two d-wide vectors, so the layer does about n² · d work. Using several attention heads at once does not remove that; see multi-head attention. More practice questions are in the large language model (LLM) interview questions list.
- An RNN reads one token at a time; a transformer attends to all tokens together.
- Transformers need a constant number of sequential steps and a path of one between any two tokens; RNNs need about n of each.
- Self-attention costs about n² · d per layer against n · d² for a recurrent layer, so it is cheaper only when n is below d.
- Transformers need positional encoding because attention has no sense of order.
- Say the cost crossover in the interview; it is the follow-up that separates levels.
Sources
- Attention Is All You Need (arXiv 1706.03762, abstract page) — arXiv (Vaswani et al.), 12 Jun 2017
- Attention Is All You Need (arXiv 1706.03762v7, full text) — arXiv (Vaswani et al.), 2 Aug 2023
- Learning long-term dependencies with gradient descent is difficult (IEEE DOI 10.1109/72.279181) — IEEE Transactions on Neural Networks (Bengio, Simard, Frasconi), 1 Mar 1994
- Learning long-term dependencies with gradient descent is difficult (Semantic Scholar record) — Semantic Scholar (Bengio, Simard, Frasconi), 1 Mar 1994
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv 2312.00752) — arXiv (Gu and Dao), 1 Dec 2023
