Seq2seq models: how encoder-decoder networks work

In this article

1. Before you start
2. Why interviewers ask this
3. Seq2seq models from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Seq2seq is the bridge between recurrent networks and the Transformer, so interviewers use it to test whether a candidate understands why attention exists. A candidate who says "the encoder encodes and the decoder decodes" has described the labels, not the design problem. The usual probes, reported in machine learning interviews, are what the context vector is, why it limits long sentences, how attention fixes that, and why the Transformer then dropped recurrence.

A weak answer names the parts. A strong answer follows one sentence through the model, shows where information is lost, and explains attention as a weighted look back at every source position. The Transformers hub places this page at the start of that story.

Seq2seq models from first principles

Seq2seq model

A seq2seq model is a neural network made of an encoder that reads an input sequence and a decoder that generates an output sequence, so that the two sequences can differ in length, as in translation or summarisation.

Context vector

A context vector is the fixed-size summary of the input that the decoder conditions on when it writes each output token; in the basic design it is the encoder's final state and does not change as the output grows.

Soft alignment

A soft alignment is a set of weights, one per source position, that says how much each source word matters for the output word being written now, and that sum to one.

In plain words

A seq2seq model works like an interpreter who listens to a whole sentence and then speaks the translation. The interpreter cannot interrupt and has no notes. For a short sentence that works. For a long one the interpreter forgets the start. Attention gives the interpreter notes. While speaking each word, the interpreter glances at the words that matter most.

The analogy breaks on memory. A person's memory fades smoothly, while a network's fixed vector has a hard size limit. In one precise sentence, a seq2seq model learns the probability of an output sequence given an input sequence by encoding the input once and decoding the output token by token (Sutskever et al., 2014).

Step 1: The encoder reads the source

The first step is the encoder, which reads the input one token at a time and updates a hidden state after each one. The input is the source tokens; the output is a hidden state that depends on everything read so far. Sutskever et al. used a deep long short-term memory (LSTM) network, a recurrent network with gated memory, with 4 layers of 1,000 cells each and 1,000-dimensional word embeddings for translation from English to French. An LSTM cell is a memory unit that keeps two numbers, a hidden state it passes on and a cell state it holds as longer-term memory.

Cho et al. described the same design: a recurrent network encodes a symbol sequence into a fixed-length vector, and a decoder is trained with it (Cho et al., 2014). Reversing the order of the source words, but not the target, raised the test score on the bilingual evaluation understudy (BLEU) translation metric from 25.9 to 30.6, because it put the first source words near the first target words. What you now know: how the source becomes a state.

Step 2: The context vector is a bottleneck

The second step hands the encoder's final state to the decoder as the context vector, and its fixed size is the model's bottleneck. The input is the last hidden state; the output is one vector of fixed size, whatever the sentence length. In Sutskever et al.'s model the deep LSTM "uses 8000 real numbers to represent a sentence", which is 4 layers times 1,000 cells times two states (the hidden state and the cell state of each cell), and that number is the same for a 3-word and a 60-word sentence.¹

Bahdanau et al. conjectured that "the use of a fixed-length vector is a bottleneck" and cited Cho et al., who showed that the performance of a basic encoder-decoder deteriorates rapidly as sentence length grows (Bahdanau et al., v7, 2016). The arithmetic in the worked example makes it concrete: 8,000 numbers is 800 per word for a 10-word sentence and 160 per word for a 50-word one.¹ What you now know: why long inputs lose information.

Step 3: The decoder writes one token at a time

The third step is the decoder, which generates the output one token at a time, conditioned on the context vector and the tokens it has already written. The input is the context vector and the previous output token; the output is a probability for every word in the target vocabulary, from which one is chosen. In Sutskever et al.'s model a softmax over 80,000 target words is computed at each step.¹

Choosing the whole sequence is a search problem. Sutskever et al. used a left-to-right beam search and report that the system "performs well even with a beam size of 1", and that a beam of size 2 gives most of the benefit of beam search (Sutskever et al., 2014); the beam search page explains the method. The loop ends when the model writes an end-of-sequence symbol. What you now know: how output grows token by token from one fixed summary.

Step 4: Attention lets the decoder look back

The fourth step replaces the single context vector with one per output word, built by attention over all the encoder states. The input is every encoder state and the decoder's previous state; the output is a weighted sum. Bahdanau et al. define the context vector for output word $i$ as $c_i={\sum }_j{\alpha }_{ij}{h}_j$, where each weight ${\alpha }_{ij}$ is a softmax over scores $e_{ij}$ from a small network that compares the decoder state with annotation $h_j$, which is Bahdanau et al.'s name for the encoder state at source position $j$ (Bahdanau et al.). Their encoder is bidirectional, so each annotation reflects the words before and after position $j$.

A toy case with four source words and made-up scores of 2.0, 0.5, -1.0 and 0.1 gives weights of 0.7030, 0.1569, 0.0350 and 0.1051, which sum to one.¹ The decoder now reads mostly the first word, and the next output word gets different weights. What you now know: the bottleneck is gone, and the weights are a soft alignment.

1 · Encode2 · Summarise3 · Decode4 · AttendReplay

Step-by-step description

1. The encoder reads the source token by token.
2. One fixed-size vector: 8,000 numbers.
3. The decoder writes one token at a time.
4. Attention replaces the single vector: each output word weighs all encoder states.

• The context vector in Sutskever et al.'s model holds 8,000 numbers at any length.
• Attention builds a weighted sum over all encoder states for each output word.

Worked example

The numbers below come from the two papers' published settings, and the arithmetic is shown in the footnote.¹

The first table decomposes the 384 million parameters that Sutskever et al. report.¹ A parameter is one learned number inside the network; training adjusts all of them, and the count measures the model's size.

The recurrent connections are 16.7 percent of the total, matching the paper's statement that 64 million of the 384 million parameters are pure recurrent connections.¹ Each LSTM layer has four gates, each with a 1,000 by 2,000 weight matrix and 1,000 biases, which gives 8,004,000 parameters per layer.

The second table compares memory, using the 8,000 numbers of Step 2.¹

The two settings are from different models, so this shows how memory scales, not a benchmark. The annotation size assumes the 1,000 forward and 1,000 backward units reported by Bahdanau et al.¹

Text description of the diagram

• A fixed vector holds 8,000 numbers at any length.
• Attention annotations hold 20,000 numbers for 10 words and 100,000 for 50 words.

These BLEU scores are Table 1 of Bahdanau et al., on all test sentences; the gains are our subtraction.¹

Text description of the diagram

• Attention raised BLEU from 13.93 to 21.50 at 30 words.
• Attention raised BLEU from 17.82 to 26.75 at 50 words.

Trade-offs and edge cases

Seq2seq with attention fixed the bottleneck but kept the recurrence, and each choice has a cost.

• Sequential computation. A recurrent encoder must process token $t$ before token $t+1$, which limits parallel training. The Transformer paper keeps the encoder-decoder layout but replaces recurrence and convolutions (an older way of mixing nearby words with small sliding filters) with attention alone, and reports better quality, more parallelisation and much shorter training (Vaswani et al., 2017); see the Transformer architecture page.
• Cost of attention. Attention keeps one state per source word, so memory and compute grow with source length, as the table shows; for self-attention variants see self-attention and multi-head attention.
• Source reversal. The reversal trick helped a unidirectional encoder; a bidirectional encoder, as in Bahdanau et al., gives each word context from both sides instead.
• Vocabulary size. A softmax over 80,000 words is a large share of the parameters, which is one reason later systems split words into pieces.
• Search cost. Beam search multiplies decoding work by the beam size, and a small beam already captures most of the gain in the original results.

How to answer it in an interview

The spoken structure is: define encoder and decoder, name the bottleneck, explain attention as a weighted sum, then state its cost and what replaced it.

Common mistakes

1. Calling the context vector the whole input. (wrong) "The decoder sees the entire source." (right) In the basic design it sees only the fixed-size summary.
2. Saying attention removes the encoder. (wrong) "With attention you don't need an encoder." (right) Attention reads the encoder's states; it does not replace them.
3. Mixing up weights and scores. (wrong) "The attention scores sum to one." (right) The scores are softmaxed; the resulting weights sum to one.
4. Claiming the Transformer is not seq2seq. (wrong) "Transformers dropped the encoder-decoder idea." (right) The original Transformer kept it and replaced recurrence with attention.
5. Blaming only training difficulty. (wrong) "Long sentences fail because the training signal fades over many steps." (right) The fixed-size vector is a separate bottleneck, which attention addresses.

Follow-up questions

What is the bottleneck in a seq2seq model?

The context vector has a fixed size however long the input is: 8,000 numbers in Sutskever et al.'s model. Bahdanau et al. conjectured that this limits long sentences, and their results show a basic encoder-decoder falling behind as length grows.

Why did Sutskever et al. reverse the source sentence?

It made the first source words close to the first target words, which shortened the distances over which the training procedure had to link a source word to its target word. In their experiments it raised test BLEU from 25.9 to 30.6, though they say they have no complete explanation.

How does attention improve on the basic encoder-decoder?

It builds a different context vector for each output word as a weighted sum of all encoder states, so information does not have to pass through one vector. In Bahdanau et al.'s tables it added 7.57 and 8.93 BLEU in the two settings above.

Why did the Transformer replace recurrent seq2seq models?

It processes all positions with attention instead of step by step, which allows more parallel training. Vaswani et al. report higher quality and much faster training than the recurrent and convolutional models they compared with.

Is a decoder-only language model a seq2seq model?

A decoder-only model generates text from a prompt without a separate encoder, so it is not the encoder-decoder form, though the generation loop is the same idea. The causal masks page compares the layouts.

Sources

1. Sequence to Sequence Learning with Neural Networks — arXiv 1409.3215 v3 (Sutskever, Vinyals, Le), 14 Dec 2014
2. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — arXiv 1406.1078 v3 (Cho et al.; EMNLP 2014), 3 Sep 2014
3. Neural Machine Translation by Jointly Learning to Align and Translate — arXiv 1409.0473 v7 (Bahdanau, Cho, Bengio; ICLR 2015), 19 May 2016
4. Attention Is All You Need — arXiv 1706.03762 v7 (Vaswani et al.), 2 Aug 2023