What is sequential data, and how do models process it?

In this article

1. Before you start
2. Why interviewers ask this
3. Sequential data from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page starts from the ordered list and builds up to the recurrent network, one idea of deep learning at a time.

Why interviewers ask this

Sequential data is why recurrent and attention models exist alongside feedforward and convolutional ones, so a question on it separates candidates who memorized architecture names from candidates who know what problem each one solves. Text, speech, sensor streams and logs are all sequences, so the topic comes up in language, forecasting and recommendation rounds.

A weak answer says "data over time" and stops. A strong answer says that order carries information, that a model must therefore use the order, and that a recurrent network does so with a hidden state and shared weights. A Staff-level answer adds what breaks: long sequences are hard to train, a fixed window forgets older steps, and a random train/test split of a time series leaks the future into training. The follow-up that decides the round is usually about memory, which is why the sections below build to it step by step.

Sequential data from first principles

Sequential data

Sequential data is data made of an ordered list of elements in which the position of an element carries information, so that reordering the elements can change their meaning or value. Sentences, audio, video, sensor readings and daily prices are sequential data. A table of unrelated customer rows is not.

Time step

A time step is the position of one element in a sequence, counted from the first element onward. In a text a time step is usually one token, in a daily series it is one day. Models index the inputs by time step, so the element at step $t$ comes after the element at step $t-1$.

Hidden state

A hidden state is a vector that a recurrent network recomputes at every time step and passes to the next step, so it holds a compressed summary of the inputs seen so far. It is called hidden because it is neither the input nor the final output of the model.

In plain words

Think of reading a recipe. You read step by step. You keep a rough picture of what is already in the bowl. That picture is your memory. If someone shuffled the lines, the recipe would break.

The analogy has a limit. Your memory is flexible. A model's memory is a fixed-size list of numbers, so it must squeeze everything into that space.

In one precise sentence, a sequence model reads inputs in order and carries a fixed-size summary of the past from step to step.

Step 1: Order carries information

Order carries information when swapping two elements changes the meaning or the value. "Man bites dog" and "dog bites man" contain the same words and mean different things. A temperature series that rises and then falls tells a different story from one that falls and then rises, even with the same readings.

Ordinary tabular data is different, because the rows are independent and shuffling them changes nothing. IBM's explainer notes that traditional networks assume inputs and outputs are independent of each other, while the output of a recurrent network depends on earlier elements of the sequence. A model that ignores order, such as one that counts words, sees both sentences above as the same input. You now know why sequences need their own models: the information sits in the order.

Step 2: The input is an ordered list indexed by time step

A sequence model takes an ordered list of vectors, one per time step. Dive into Deep Learning describes the inputs as an ordered list of feature vectors, each one indexed by a time step. Write them $x_1$ to $x_T$, where $T$ is the length, and the length can differ from one example to the next. Inputs x1 to x3 enter in order, indexed by time step t.

The sequence can sit on either side of a task. As the book's recurrent networks chapter puts it, "A great many learning tasks require dealing with sequential data." Time series prediction and video analysis take a sequence in and give one value out. Image captioning and speech synthesis produce a sequence. Translation and dialogue need a sequence on both sides.

Text description of the diagram

• Time series prediction takes a sequence in and gives one value out.
• Translation and dialogue need a sequence on both sides.

A simple way to use the order is a window: feed the last few values as features and ask for the next one. A model that forecasts a series from its own earlier values is called autoregressive. For text, a language model is a model that estimates the joint probability of a whole sequence of tokens, as the same book describes it.

Step 3: A hidden state carries memory from step to step

A recurrent neural network (RNN) is a neural network that reads a sequence one element at a time and passes a hidden state from each step to the next. At step $t$ it combines the new input $x_t$ with the previous state $h_{t-1}$ to make the new state $h_t$, written $h_t=f(h_{t-1},x_t)$ for some learned function $f$.

IBM describes the hidden state as a memory that stores information about previous inputs and tracks context at each time step. Input: the current element and the old state. Output: the new state, and optionally a prediction. Unrolled over three steps, the hidden state passes from h0 to h1, h2 and h3 from left to right.

This answers the follow-up from the opening. The model remembers earlier steps only through the hidden state, a fixed-size vector that every step overwrites. Information that does not survive that squeeze is lost. Gated cells such as the long short-term memory (LSTM) cell are built to control what survives; see LSTM explained. You now know where the memory lives and why it is limited.

Step 4: The same weights are reused at every step

Parameter sharing is the reuse of the same weights at every time step, so a model with a fixed number of parameters can read a sequence of any length. Dive into Deep Learning states that recurrent networks are unrolled across time steps with the same underlying parameters applied at each step. The same weights W are used in every cell.

Goodfellow, Bengio and Courville explain the reason: with separate parameters for each time index, a model could not generalize to sequence lengths not seen during training, nor share statistical strength across positions. Unrolling also gives a view as a feedforward network, which is how training by backpropagation is applied through time, called backpropagation through time. The worked example below counts the parameters.

1 · Ordered inputs2 · Time steps3 · Hidden state4 · Shared weightsReplay

Step-by-step description

1. The tokens appear in order.
2. Each input has a time step t.
3. The hidden state passes along.
4. The same weights W sit in every cell.

• Inputs x1 to x3 enter in order, indexed by time step t.
• The hidden state passes from h0 to h1, h2 and h3 from left to right.
• The same weights W are used in every cell.

Worked example

Five distinct tokens can be arranged in 120 orders, yet a model that only counts tokens gives the same answer for every one of them.¹ That is the shuffle test from Step 1 in numbers: counting keeps one distinct input where an order-aware view keeps 120.¹

Now a toy recurrence with illustrative numbers. Let the state update be $h_t=0.5h_{t-1}+x_t$ with $h_0=0$, and feed the inputs 1, 2, 3. The states are 1, 2.5 and 4.25. Feed the same inputs reversed, 3, 2, 1, and the states are 3, 3.5 and 2.75.¹ The same three inputs end at hidden state 4.25 in one order and 2.75 in the reverse order. Reversing the order changes the final hidden state even though the inputs are identical. The sum of the inputs is 6 both times, so a model that sees only the total cannot tell them apart.¹ The weight 0.5 is the same at every step, which is parameter sharing.

Text description of the diagram

• The same three inputs end at hidden state 4.25 in one order and 2.75 in the reverse order.
• Reversing the order changes the final hidden state even though the inputs are identical.

Next a window forecast on an illustrative seven-day temperature series: 18, 19, 21, 20, 22, 23, 25. A window of 3 days gives 4 training examples, such as (18, 19, 21) with target 20.¹ Forecasting with the mean of the window has a mean absolute error (MAE) of 2.0, and forecasting with the last value has an MAE of 1.5.¹ On a rising series the recent value beats the older average, which shows the order matters even inside a window.

Finally the parameters. An RNN cell with an input of size 3 and a hidden state of size 4 has 4 × 4 + 4 × 3 + 4 = 32 parameters, whatever the length.¹ With separate weights for each time step, a sequence of 5 steps needs 160 parameters and a sequence of 1,000 steps needs 32,000.¹

Trade-offs and edge cases

Each way of reading a sequence trades memory length, cost and simplicity.

Long sequences are the main difficulty for recurrent training, because gradients pass through many steps and can vanish or explode; the problem is covered in exploding gradients. For text, attention has largely taken the place of recurrence in current models. Self-attention, the operation inside a transformer, compares every position with every other position and has no sense of order by itself, so Vaswani et al. (2017, section 3.5) add information about the position of each token, called positional encodings. The hidden-state idea survives in gated recurrent cells such as the LSTM and in newer models that also carry a state along the sequence. A convolutional network is the usual choice when only local patterns matter.

Edge cases: sequences have different lengths, which shared weights allow (Goodfellow et al. note that most recurrent networks process variable length). Time series should be split by time, with training on the past and testing on the future, because a random train/test split leaks future information into training. Shuffling whole sequences or training windows between batches is still fine. And not every ordered thing is sequential in this sense: if the order is arbitrary, such as a list sorted by customer id, the order carries no information.

How to answer it in an interview

The spoken structure: define the term, name the memory, say why shared weights allow any length, then give the trade-off.

Common mistakes

• "Sequential data means time series." (wrong) Text, audio and video are sequences too, and the book treats them together. (right) "A time series is one kind of sequential data."
• "An RNN has different weights at each step." (wrong) The same weights are applied at each step. (right) "The weights are shared across time steps, and the state changes."
• "Sequences must all have the same length." (wrong) Shared weights let one model read different lengths. (right) "The length can vary; the parameter count stays fixed."
• "A random train/test split is fine for a time series." (wrong) It puts future values in the training set. (right) "I split by time: train on the past, test on the future."

Follow-up questions

Is a time series the same as sequential data?

No. A time series is sequential data indexed by time, such as daily temperatures. Sentences and audio are sequential data too, with position taking the place of time. Forecasting a series from its own past values is where probability and statistics ideas such as autocorrelation, the way a series relates to its own earlier values, come in.

Why can't you shuffle sequential data?

Sequential data cannot be shuffled inside a sequence, because the order is the information; in the worked example the same inputs end at 4.25 or 2.75 depending on order.¹ Whole sequences or training windows can still be shuffled between batches. For a time series, the train/test split must follow time, because a random split puts future values in the training set.

How do recurrent networks handle sequences of different lengths?

Recurrent networks apply the same weights at every step, so the model reads as many steps as the sequence has. Separate weights per step would tie the model to one length.

Can a convolutional network process sequential data?

Yes. A one-dimensional convolution also shares parameters along the sequence, but each output depends on a small number of neighbours. See convolutional networks.

What goes wrong when you train on long sequences?

Gradients must travel through many steps, so they can explode or vanish, and training becomes unstable or forgets early inputs. Gated cells and gradient clipping are the usual remedies; see LSTM and exploding gradients. More practice questions are in deep learning interview questions.

Sources

1. Dive into Deep Learning, chapter on recurrent neural networks (arXiv 2106.11342, v1) — d2l.ai (Zhang, Lipton, Li, Smola), 21 Jun 2021
2. Dive into Deep Learning: Working with Sequences — d2l.ai (Zhang, Lipton, Li, Smola), 21 Jun 2021
3. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
4. Deep Learning, chapter 10: Sequence Modeling: Recurrent and Recursive Nets — MIT Press (Goodfellow, Bengio, Courville, 2016), 1 Nov 2016
5. What is a recurrent neural network? — IBM Think, 26 Oct 2026