In this article
Before you start
This builds on the transformer architecture and on neural networks.
Why interviewers ask this
A candidate who says "the model predicts the next word" has described the loop but not the choice, and the choice is where the interesting questions start. Interviewers ask this to see whether you separate three things: what the network computes (a score for every token), what a decoding rule does with those scores (picks one), and what repeats (the loop). Reported in interviews at AI labs and large tech companies, it often opens a longer thread on sampling, latency and why outputs vary.
A weak answer says the model "picks the most likely word." A strong answer names tokens, logits, probabilities and the decoding rule. A Staff-level answer adds the trade-off between repeatable and varied output, and explains why the loop makes generation slow: each new token needs another pass.
How a new token is generated, from first principles
- Decoding rule
- A decoding rule is the procedure that chooses one token, or one whole sequence, from the model's probability distribution, such as greedy search, sampling or beam search. The model supplies the probabilities, and the rule makes the choice.
- Autoregressive generation
- Autoregressive generation is a way of producing a sequence in which each new token is chosen using all the tokens produced so far and then appended to them. The output grows one token at a time, so an early mistake carries into every later step.
In plain words
Think of a phone keyboard that suggests the next word. It scores many words and shows the best few. You tap one and it enters the text. A language model works the same way. A rule does the tapping, and the model scores pieces of words. The analogy breaks here: a keyboard shows a few suggestions, while the model scores every token it knows at every step.
A token is a chunk of text that the model treats as one unit: according to the Hugging Face generation blog (accessed 2026-10-26), it may be a whole word, a sub-word or a single character, depending on the model.
In one precise sentence, a large language model (LLM) generates text by repeatedly mapping the tokens so far to a probability distribution over its vocabulary, selecting one token from it and appending that token.
Step 1: The model reads every token so far
The model takes the prompt plus any text it has already produced as one list of tokens and first looks up each token ID in an embedding table, which gives one vector per token, and adds position information so the order is known. It then runs those vectors through its stacked transformer layers, which produces one output vector per position. Self-attention lets each position use earlier tokens, and multi-head attention runs several such views side by side. The Hugging Face blog (accessed 2026-10-26) explains that the logits at one position describe the next token as if the input ended there, and that only the last position matters when a new token is needed.
Input: the tokens so far. Output: a score vector at each position, of which only the last is used. You now know the model produces scores, not words.
Step 2: Scores become probabilities
At the last position a final linear layer, called the language-model head or unembedding, multiplies the output vector by a matrix with one column per vocabulary token, which gives one logit per token in its vocabulary, the full list of tokens it knows. The original transformer of Vaswani et al. (2017) shares "the same weight matrix between the two embedding layers and the pre-softmax linear transformation". A softmax turns the logits into probabilities that sum to one. Take the prompt "The cat sat on the" with a toy vocabulary of five tokens whose logits are 3.0 (mat), 2.0 (floor), 1.0 (sofa), 0.0 (roof) and -1.0 (moon), all illustrative. The softmax gives 0.6364, 0.2341, 0.0861, 0.0317 and 0.0117.1
A real model does this over its whole vocabulary, not over five tokens; the Hugging Face generation blog (accessed 2026-10-26) describes the next step as selecting the next token from these logits. Input: logits. Output: probabilities. Nothing has been chosen yet.
Step 3: A decoding rule picks one token
The decoding rule chooses the token, and different rules give different tokens from the same probabilities. Greedy search takes the highest probability; the Hugging Face tutorial (accessed 2026-10-26) says it is the default decoding strategy in generate() unless a model's generation config says otherwise. In the toy example, greedy picks mat, which has probability 0.6364.1
Sampling draws a token at random in proportion to its probability, which the library enables with do_sample=True; in the toy example mat is drawn with probability 0.6364 and floor with 0.2341.1 Beam search keeps several candidate sequences and chooses the highest-scoring one among the beams it keeps (often length-normalised), so it can begin with a token greedy search would skip; it is an approximation, not an exhaustive search (Hugging Face, text generation strategies, accessed October 2026).
Input: probabilities. Output: one token. Run-to-run variation comes mainly from this step, not from the network.
Step 4: The token is appended and the loop repeats
The chosen token joins the input and the whole procedure runs again, which is why the process is called autoregressive. Decoder-only models, such as those in the generative pre-trained transformer (GPT) family covered in what GPT stands for, generate this way. Because later tokens depend on earlier ones, a wrong token is built on by every later step; the Hugging Face blog (accessed 2026-10-26) notes that as soon as the assistant gets a token wrong, all subsequent candidates must be invalidated.
Generation ends when a stopping rule fires; the tutorial (accessed 2026-10-26) lets you supply custom stopping criteria. A common setup is a special end-of-sequence (EOS) token or a length limit. Implementations also keep earlier attention keys and values in a key-value (KV) cache, so each pass processes mainly the newest token; that saves time and does not change the choice.
Step-by-step description
- The model reads all tokens so far.
- Logits become softmax probabilities.
- A decoding rule picks one token.
- The token is appended and fed back in.
- A language model scores every vocabulary token for the next position.
- A decoding rule picks one token, which is appended before the loop repeats.
How do temperature, top-k and top-p change the pick?
Temperature is a number that divides the logits before the softmax: below one it sharpens the distribution and above one it flattens it. The Hugging Face tutorial (accessed 2026-10-26) lists temperature as an option that requires do_sample=True, with a rule of thumb of high values for creative tasks and low values for tasks that need "thinking". It reshapes the choice, not the model.
| Temperature | mat | floor | sofa | roof | moon |
|---|---|---|---|---|---|
| 0.5 | 0.8647 | 0.1170 | 0.0158 | 0.0021 | 0.0003 |
| 1.0 | 0.6364 | 0.2341 | 0.0861 | 0.0317 | 0.0117 |
| 1.5 | 0.5046 | 0.2591 | 0.1330 | 0.0683 | 0.0351 |
Top-k keeps only the k most likely tokens and renormalises them: with k equal to 2, mat becomes 0.7311 and floor 0.2689. Top-p, introduced as nucleus sampling by Holtzman et al. (2019), keeps the smallest set of tokens whose probabilities add up to at least p: with p equal to 0.9 it keeps mat, floor and sofa (cumulative 0.9567) and rescales them to 0.6652, 0.2447 and 0.0900.1 The library applies such reshaping through logits processors, per the tutorial.
Text description of the diagram
- At temperature 0.5 the token mat has probability 0.8647.
- At temperature 1.5 the token mat has probability 0.5046.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
From a high level point of view, how is a new token generated?
I describe it as a loop that runs once per token. The model looks up embeddings for all tokens so far and runs them through its layers; a final linear layer maps the last hidden vector to a score, called a logit, for every vocabulary token. A softmax turns those scores into probabilities.
Then a decoding rule picks one token: greedy takes the highest, sampling draws from the distribution, and beam search keeps several candidate sequences. Temperature, top-k and top-p reshape that choice. The chosen token is appended, the model runs again, and the loop stops at an end-of-sequence token or a length limit. The trade-off is that greedy is repeatable but can get stuck, while sampling is more varied but less predictable.
What makes it Staff-level: it separates scoring from choosing, names the trade-off between repeatable and varied output, and says where the loop stops. The spoken structure is: loop, scores, probabilities, decoding rule, append, stop.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "The model predicts the next word." | Names tokens, logits, probabilities and a pick. | Also states that the loop appends the pick and repeats. |
| Depth | Skips how the pick is made. | Names greedy and sampling. | Adds beam search, temperature, top-k and top-p. |
| Trade-offs | None given. | Says greedy is repeatable, sampling varies. | Links the choice to quality, diversity and repeated runs. |
| Communication | Rambles through layers. | Walks the loop in order. | Gives the loop in about a minute, then offers depth. |
| Follow-up readiness | Stuck on "why not always the top token?" | Explains the greedy trap. | Explains the cache, stopping and context limits. |
Three variants of this question
Does the model always pick the most likely token?
No. Only greedy search always does; sampling draws in proportion to probability, and beam search compares whole sequences. In an illustrative two-step case, token A has probability 0.5 and B has 0.4; after A the best next token x has 0.3, and after B the best next token y has 0.9. Greedy gives A then x with probability 0.15, while beam search keeping both finds B then y with 0.36.1
Why does the model generate tokens rather than words?
Because the model's vocabulary is made of tokens, which may be words, sub-words or single characters, depending on the model, per the Hugging Face blog (accessed 2026-10-26). Sub-words let a fixed vocabulary spell any text; see byte pair encoding.
How does the model know when to stop?
The network does not decide; a stopping rule does. Common rules are a length limit or a special end-of-sequence token, and the library lets you supply custom stopping criteria (tutorial, accessed 2026-10-26).
Common mistakes
- "The model writes the whole sentence at once." (wrong) It produces one token at a time, and each depends on the earlier ones. (right) "It generates autoregressively: choose a token, append it, run again."
- "The model picks a word." (wrong) The unit is a token, which may be a sub-word or a character. (right) "It picks a token from its vocabulary."
- "The model always picks the most likely token." (wrong) That is true only for greedy decoding. (right) "Greedy takes the top token; sampling and beam search choose differently."
- "Greedy gives the best sentence because each step is best." (wrong) A locally best token can lead to a weaker whole sequence, as the beam example shows. (right) "Greedy is simple and repeatable; beam search can find a higher-probability sequence."
- "Temperature changes the model." (wrong) It only reshapes the probabilities used for selection and applies when sampling. (right) "Temperature rescales the logits before the softmax; the weights stay the same."
Follow-up questions
Why is generation slow, and what does the KV cache save?
Each new token needs another pass because it depends on the tokens before it, so the passes cannot be done together for one answer. The KV cache stores earlier keys and values so the pass mainly processes the newest token. For the attention cost behind this, see why attention is quadratic.
What happens when the context window is full?
The model can only read a limited number of tokens, so older text must be cut or summarised before generation continues. The context window page covers what that limit costs.
How does generation differ in an encoder-decoder model?
The loop is the same, one token at a time, but the input is encoded once and each step also attends to that encoding. See seq2seq models.
What happens to the output as temperature moves toward zero or grows large?
Toward zero, the distribution concentrates on the top token, so sampling behaves like greedy search. As it grows, the distribution flattens and unlikely tokens are drawn more often. Practise more of these in the large language model interview questions and natural language processing interview questions.
- A new token is generated by scoring every vocabulary token, converting scores to probabilities and picking one.
- The network computes logits; a decoding rule chooses the token.
- Greedy is repeatable, sampling is varied, and beam search compares whole sequences.
- Temperature, top-k and top-p reshape the distribution before the choice.
- The chosen token is appended and the loop repeats until a stopping rule fires.
Sources
- Text generation strategies (Transformers, main branch) — Hugging Face (documentation; the page shows no publication date, so the date is the read date), 26 Oct 2026
- Text generation (LLM tutorial, Transformers main) — Hugging Face (documentation, accessed date), 26 Oct 2026
- Assisted Generation: a new direction toward low-latency text generation — Hugging Face (blog, accessed date), 26 Oct 2026
- Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
- The Curious Case of Neural Text Degeneration (arXiv 1904.09751) — arXiv (Holtzman et al.), 22 Apr 2019
