LSTM explained: how a long short-term memory cell works

In this article

1. Before you start
2. Why interviewers ask this
3. LSTM from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

An LSTM question tests whether you understand why a design exists and not only what its boxes are called. Many candidates can draw the three gates and cannot say what breaks without them. The usual probes are "what problem does the forget gate solve?", "why is the cell state separate from the hidden state?" and "why did transformers replace LSTMs?". A weak answer lists the gate names. A strong answer says that the cell state is updated by addition, so the training signal can pass through many steps without being squashed at every one, and that the forget gate controls how much survives. It then says what the transformer changed.

LSTM from first principles

Cell state

The cell state is the vector of numbers an LSTM carries from one step to the next as its long-term memory, changed only by a gated erase and a gated write instead of being rebuilt from scratch at each step.

Gate

A gate is a vector of numbers between 0 and 1, produced by a sigmoid, that is multiplied element by element with another vector, so that 0 blocks a value, 1 lets it through and numbers between scale it.

Forget gate

The forget gate is the gate that multiplies the old cell state by a number between 0 and 1 for each memory slot, deciding how much of each stored value survives to the next step.

Vanishing gradient

A vanishing gradient is a training signal that shrinks toward zero as it is passed backward through many steps, so the weights that acted early in a sequence receive almost no correction.

In plain words

An LSTM is like a notebook with an eraser and a pencil. At each step you first erase some of what is written. Then you write something new. Then you read a summary aloud. Three small controllers decide how hard to erase, how much to write and how much to read. They are the gates, and they are learned from data. The analogy breaks in one place: a gate is a fixed formula, not a judgment. In one precise sentence, an LSTM updates its cell state as the old state times a forget gate plus a new candidate times an input gate, and outputs a hidden state that is the output gate times the tanh of the cell state.

A recurrent neural network (RNN) is a network that reads a sequence one item at a time and passes a summary, the hidden state, from each step to the next. The hidden state is the cell's short-term output at each step, the thing the next layer or the next step reads.

Why do plain recurrent networks forget?

Plain recurrent networks forget because the training signal is multiplied by a number at every step on its way backward. Hochreiter and Schmidhuber (1997) describe how error signals flowing back through time tend to blow up or vanish, and how the change depends exponentially on the size of the weights (Hochreiter and Schmidhuber, 1997, section 1). Pascanu, Mikolov and Bengio (2013) analyse the same two problems and propose gradient clipping for the exploding case (Pascanu et al., 2013). If each step multiplies the signal by 0.9, only 12.16% is left after 20 steps and 0.0027% after 100.¹ The words at the start of a long sentence simply stop mattering to the weights.

Step 1: The forget gate decides what stays

The forget gate looks at the new input and the previous hidden state and outputs a number between 0 and 1 for every slot of the cell state, and the old state is multiplied by these numbers. Near 1 keeps a slot; near 0 erases it. Gers, Schmidhuber and Cummins (2000) added it, absent from the 1997 design, so a cell can learn to reset itself, as the state may otherwise grow without bound on a continual stream (Gers et al., 2000).

At a gate of 0.5, which is what a zero bias gives, the signal falls below 1% after 6.6 steps. At 0.9933 (a bias of 5) it takes 685.8 steps.¹ The start depends on the library: Keras adds 1 to the forget bias by default, so the gate starts near 0.73 (Keras LSTM documentation, read 2026-10-24), and PyTorch draws all weights and biases at random, so its gates start scattered around 0.5 (PyTorch nn.LSTM, read 2026-10-24). What you now know: the forget gate sets the memory length, and its start matters.

Step 2: The input gate and the candidate write new information

The candidate is a vector of proposed new values, computed with tanh from the input and the previous hidden state. The input gate then scales it. Each slot gets its own number between 0 and 1, so the cell can write to some slots and leave others alone. The candidate is bounded between -1 and 1, which keeps one step from pushing the memory to extreme values. The PyTorch documentation gives the formulas: the gate is a sigmoid of a weighted sum of the input and the last hidden state, and the candidate is a tanh of a similar sum (PyTorch nn.LSTM, read 2026-10-24). What you now know: writing is a two-part act, a proposal and a permission.

Step 3: Update the cell state by adding

The new cell state is the old state times the forget gate, plus the candidate times the input gate. The operation that joins the two parts is an addition, and that is the central design choice. Along the cell state, the derivative with respect to the previous state is just the forget gate, with no weight matrix and no squashing curve in the way. Hochreiter and Schmidhuber call the original form of this path a constant error carousel, and say it enforces constant error flow (1997 abstract). With the gate near 1, the signal passes almost untouched. What you now know: addition lets the signal travel; multiplication by a weight matrix at every step is what kills it.

Step 4: The output gate reads the memory out

The hidden state is the output gate times the tanh of the cell state. The output gate decides how much of the memory to expose at this step, and the tanh squeezes the memory into the range -1 to 1. The cell state itself is not shown to the next layer. Only the hidden state is. This separation lets the cell keep information that is not useful right now but will be needed later. The tanh keeps every exposed value in a fixed, small range. What you now know: the cell state is private memory, and the hidden state is what the rest of the network sees.

1 · Forget2 · Write3 · Update4 · ReadReplay

Step-by-step description

1. The forget gate scales the old cell state.
2. The input gate scales the candidate.
3. Kept memory plus written memory.
4. The output gate reads out the hidden state.

• The forget gate scales the old cell state.
• The input gate scales the candidate that is written.
• The new cell state is the kept memory plus the written memory.
• The output gate reads out the hidden state.

Worked example

Two small computations show what a gate does and how long a signal lives. Every number comes from a short script that was run, with its output saved.¹ The first uses made-up weights, so it is illustrative and not a trained model. The second uses the formula that the signal along the cell state is multiplied by the forget gate at each step, and ignores the smaller paths through the other gates.

The cell has one input and one hidden unit, with an illustrative input of 1.0. The previous hidden state is 0.0 and the previous cell state is 2.0. The weights are chosen so that the gates take plain values.

The memory fell only from 2.0 to 1.9362 because the write nearly replaced what the forget gate removed. The next table asks the second question, how many steps a training signal survives for different forget gates, measured as the number of steps until less than 1% is left, which is steps = ln 0.01 / ln f for a per-step factor f. The bias labels assume that the weighted inputs are near 0, so the gate is the sigmoid of the bias alone.¹

Two readings matter. First, with a zero bias the forget gate starts near 0.5, and in this illustration, where the plain network's factor is an assumed 0.9, a gate of 0.5 is worse than that plain network because it halves the signal at every step. Second, a bias of 1 is not enough to reach long memory by itself: the gate is 0.7311, which still loses the signal in 14.7 steps. What Jozefowicz, Zaremba and Sutskever (2015) report is the practical effect: adding a bias of 1 to the forget gate closes the gap between the LSTM and the gated recurrent unit on their tasks (Jozefowicz et al., 2015).

A bias of 1 gives the network a start that it can then push higher, since the gate is learned.

Text description of the diagram

• A plain recurrent network with weight 0.9 keeps the signal above 1% for 43.7 steps.
• A forget gate of 0.5 keeps it for 6.6 steps and a gate of 0.9933 for 685.8 steps.

Trade-offs and edge cases

An LSTM has four times the parameters of a plain recurrent layer of the same size, and it still reads a sequence one step at a time. For an input of size 128 and a hidden size of 256, the PyTorch layout gives 395,264 parameters for the LSTM, 296,448 for a gated recurrent unit (GRU) and 98,816 for a plain recurrent layer, and the library's own count matched.¹ The GRU is the simpler gated design that Cho et al. (2014) introduced for translation (Cho et al., 2014): it merges the gates and drops the separate cell state, which is why it has three quarters of the parameters.

Vaswani et al. (2017) built the transformer on attention alone, "dispensing with recurrence", and reported better parallelism and shorter training times (Vaswani et al., 2017). A recurrent layer must compute step 500 before step 501, so its 1,000 steps are sequential, while a transformer layer compares all 1,000 tokens in one pass at the cost of 1,000,000 token pairs (500,500 with a causal mask).¹ That holds in training and when reading a whole known input; generating new tokens is still one token at a time, with the keys and values of earlier tokens cached. See self-attention and the transformer for that side.

Text description of the diagram

• At input 128 and hidden 256 an LSTM has 395,264 parameters and a GRU has 296,448.
• A recurrent layer runs 1,000 steps one after another for 1,000 tokens.

Practice is not settled on whether recurrence is finished. Beck et al. (2024) report that their extended LSTM, with exponential gating and a matrix memory, compares favorably with transformers and state-space models, and that is the authors' claim for their own benchmarks (Beck et al., 2024). Two edge cases follow from the arithmetic above. A forget gate stuck near 0 gives a network that forgets everything, and a gate stuck near 1 never clears the memory, which is the continual-stream failure that Gers et al. describe. See also the neighbouring page on the transformer architecture.

How to answer it in an interview

Say it in this order: the cell state, the three gates, why addition protects the signal, then the cost.

Common mistakes

"The LSTM has no vanishing gradient."

(Wrong) "LSTMs solve the vanishing gradient." The signal along the cell state is multiplied by the forget gate, and a gate of 0.5 loses the signal in 6.6 steps in the worked example. (Right) "LSTMs make long memory possible; the forget gate has to learn to stay near 1."

"The cell state and the hidden state are the same thing."

(Wrong) "The cell state is the output." Only the hidden state, the output gate times tanh of the cell state, leaves the cell. (Right) "The cell state is internal memory, and the hidden state is its gated, squashed view."

"The original LSTM had input, forget and output gates."

(Wrong) "The original LSTM already had a forget gate." The 1997 paper has multiplicative gate units, and the forget gate came in the 2000 paper of Gers et al. (Right) "The forget gate was added later and is now standard."

"A bias of 1 on the forget gate gives long memory."

(Wrong) "Initialize the bias to 1 and the memory is long." A gate of 0.7311 still loses 99% of the signal in 14.7 steps. (Right) "A bias of 1 is a better starting point than 0, and training has to push the gate toward 1 where memory is needed."

Follow-up questions

Is LSTM still relevant?

Yes in some settings, though not at the top of language modeling. Transformers replaced recurrent networks in many sequence tasks after 2017, and research such as xLSTM tests whether scaled-up recurrence can compete again, with results so far reported by its authors.

LSTM vs transformer: what is the difference?

An LSTM compresses the past into a fixed-size state and reads one step at a time. A transformer keeps keys and values for every earlier token and compares all of them in one pass when reading a known input (generation is still token by token), at a cost that grows with the square of the length.

LSTM vs GRU: which should you use?

Start with the one your library tunes well, since tasks differ. The GRU has three quarters of the parameters at the same size, 296,448 against 395,264 at input 128 and hidden 256, and Jozefowicz et al. found neither wins everywhere.

Why initialize the forget gate bias to 1?

A bias of 0 gives a gate near 0.5, which halves the signal at every step. A bias of 1 gives 0.7311 (the Keras default adds it), and Jozefowicz et al. report that it closes the gap between the LSTM and the GRU.

What if the forget gate is stuck near 0?

Then the cell state is erased at every step and the model behaves like a network with no long memory. Raising the forget bias or checking for a learning-rate problem are the usual first checks.

Sources

1. Long Short-Term Memory (Neural Computation 9(8), 1735-1780) — Hochreiter and Schmidhuber, MIT Press (issue date 15 Nov 1997; read 2026-10-24), 15 Nov 1997
2. Learning to Forget: Continual Prediction with LSTM (Neural Computation 12(10), 2451-2471) — Gers, Schmidhuber and Cummins (issue date 1 Oct 2000; read 2026-10-24), 1 Oct 2000
3. On the difficulty of training Recurrent Neural Networks — arXiv (Pascanu, Mikolov, Bengio), v2, 16 Feb 2013
4. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — arXiv (Cho et al.), v3, 3 Sep 2014
5. An Empirical Exploration of Recurrent Network Architectures — PMLR 37, ICML 2015 (Jozefowicz, Zaremba, Sutskever), 1 Jun 2015
6. Attention Is All You Need — arXiv (Vaswani et al.), v7, 2 Aug 2023
7. xLSTM: Extended Long Short-Term Memory — arXiv (Beck et al.), v2, 6 Dec 2024
8. LSTM layer (Keras documentation) — Keras (date = read date), 24 Oct 2026
9. torch.nn.LSTM (PyTorch documentation 2.14) — PyTorch (date = read date), 24 Oct 2026