How an LLM picks the next word, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. How an LLM picks the next word from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits inside the LLM inference guide and builds on how a transformer produces its scores.

Why interviewers ask this

A candidate who cannot explain decoding treats a language model as a black box that "just answers." Interviewers ask about it because the same model produces dull, repetitive text or wandering nonsense depending on a few settings, and an engineer must know which knob fixes which problem. Holtzman et al. show that decoding choices alone can change text quality dramatically, even with exactly the same model.

Candidates report being asked this in many forms: the difference between top-k and top-p, what temperature zero means, how to adjust creativity, and how to stop repeated phrases. A weak answer lists names. A strong answer says what each setting does to the probability list. A Staff-level answer also says what each one cannot fix and where the settings interact. For example, temperature changes which tokens fall inside top-p's set, because top-p is applied after temperature in the usual order.

How an LLM picks the next word from first principles

Decoding strategy

A decoding strategy is the rule that turns a language model's next-token probabilities into one chosen token at each step, repeated until the text ends. The model stays the same; only the choosing rule changes.

In plain words

Imagine a person finishing your sentence at a party. In their head, some endings feel very likely and many feel unlikely. They do not say the single likeliest ending every time, or the conversation would be boring; they also would not say a random word. Decoding is the habit they use to choose. The analogy breaks in one place: a person chooses from feeling, while a model chooses by a fixed arithmetic rule you can change.

In one precise sentence, a language model produces a probability for every token in its vocabulary, and the decoding strategy selects one token from that list, again and again, with each chosen token fed back in as input. This repeated sampling of the next word is how Conformal Nucleus Sampling describes generation. Below, a toy list of six tokens shows each stage; the numbers are illustrative.¹

1 · Scores2 · Temperature3 · Trim4 · ChooseReplay

Step-by-step description

1. Softmax turns scores into probabilities.
2. Low sharpens, high flattens.
3. Top-k or top-p cuts the tail.
4. One token is chosen, then repeat.

• The model scores every token and softmax makes probabilities.
• Temperature reshapes the probabilities.
• Top-k and top-p cut the unreliable tail.
• One token is chosen, then the loop repeats.

Step 1: The model scores every token and softmax makes probabilities

At each step the model outputs one logit per token in its vocabulary, and softmax turns the logits into probabilities that sum to one. The logits come from the final layer of the transformer after it reads all tokens so far.

Input: the text so far. Output: one probability per possible next token. In the peaked toy case (illustrative), the token "the" gets 0.8246 and "sat" gets 0.002; in the flat toy case, the best token gets only 0.2308.¹ Notice that the model has not chosen anything yet: it has only produced a list. You now know what the later steps act on, and why a confident model and an unsure model give very different lists to the same rule.

Step 2: Temperature reshapes the probabilities

Temperature divides every logit by a number T before softmax, so a low T widens the gap between tokens and a high T narrows it. With logits $z_i$, the probability of token $i$ is:

In the peaked toy case the top token's probability is 0.9793 at T of 0.5, 0.8246 at 1.0 and 0.6569 at 1.5; in the flat case it is 0.3046, 0.2308 and 0.2082.¹ The Hugging Face documentation lists a default of 1.0, which leaves the model's own probabilities unchanged. Because softmax of z divided by T is proportional to $p_i^{1/T}$, dividing the logits is the same as reshaping the probabilities directly. Temperature adds no knowledge; it only reshapes the same list.

As T shrinks, the top token takes almost everything: at T of 0.1 the peaked toy case gives it 1.0 to four places.¹ That limit is called greedy decoding: greedy decoding is a rule that always takes the highest-probability token. Exactly zero cannot be computed with this formula, so software normally treats it as a special case that takes the top token directly.

Step 3: Top-k and top-p cut the unreliable tail

Top-k and top-p remove unlikely tokens before the draw, because a long tail of poor options adds up to a real chance of a bad pick. Top-k keeps a fixed count; the Hugging Face documentation lists a default k of 50. Top-p, introduced by Holtzman et al., keeps the smallest set of most probable tokens whose probabilities add up to p or higher, the same definition the Hugging Face documentation gives, so its pool adapts to the model's certainty. These defaults apply only when sampling is turned on (do_sample=True); otherwise generate() decodes greedily.

Top-k with k of 3 keeps 3 tokens in both toy cases, while top-p at 0.9 keeps 2 tokens when peaked and 6 when flat. The flat case shows the weakness of a fixed k: it discards 41% of the mass the model considers plausible.¹ After trimming, the remaining probabilities are rescaled to sum to one. You now know why top-p is the more flexible rule.

Text description of the diagram

• Top-k with k of 3 keeps 3 tokens in both toy cases.
• Top-p at 0.9 keeps 2 tokens when peaked and 6 when flat.

Step 4: One token is chosen, then the loop repeats

The last stage chooses one token from the trimmed list, either by taking the top one or by a random draw weighted by the probabilities. The chosen token joins the text and the model runs again, reusing the key-value (KV) cache; servers interleave many loops with continuous batching.

Greedy decoding takes the top token every time, so a locally best token can mislead. Beam search is a search rule that keeps the w best partial sequences at each step. In a two-step toy (illustrative), greedy takes A with probability 0.6 and then a 0.5 continuation, for 0.3, while beam search with width 2 also keeps B and finds B then z at 0.36.¹

Beam search can still loop: a 2025 International Conference on Computational Linguistics (COLING) study of decoding hyperparameters reports lower diversity as the beam widens, from 12.78 at width 3 to 6.17 at width 50. Beam search and greedy decoding suit tasks with one right answer, such as translation or structured output; sampling suits open-ended text, where maximizing likelihood gives bland, repetitive output (Holtzman et al.).

The Staff-level answer

What makes this Staff-level is the last move: it names the failure of the obvious rule, and it treats the settings as separate levers on one distribution rather than as a list of names.

The rubric

Three variants of this question

What is the difference between top-k and top-p (nucleus) sampling?

Top-k keeps a fixed number of tokens, while top-p keeps the smallest set whose probabilities reach a threshold, so the set size changes with the model's confidence. In the toy flat case top-k with k of 3 keeps 3 tokens and loses 41% of the mass, while top-p at 0.9 keeps all 6.¹

When you set the temperature to 0.0, which decoding strategy are you using?

You are effectively using greedy decoding, because as temperature falls toward zero the top token takes almost all the probability. Say that exactly zero is usually handled as a special case, since the formula would divide by zero. For a Staff-level nuance, do not call it perfectly deterministic: on shared hardware, floating-point rounding and batching can change which of two near-tied tokens wins, so two runs may occasionally differ. Check your provider's documentation.

How do you adjust the creativity of generative AI models?

Raise temperature or widen top-p or top-k for more varied wording, and lower them for steadier output. Temperature only reshapes probabilities; it adds no knowledge, so "creative" output can also be wrong. Change one setting at a time and compare outputs.

Common mistakes

• "The model picks the most likely word." (wrong) The usual rule samples from a trimmed distribution, and always taking the top token gives flat, repetitive text. (right) "The model produces probabilities, and the decoding strategy chooses from them."
• "Beam search fixes repetition." (wrong) Holtzman et al. report that using likelihood as the decoding objective leads to bland, strangely repetitive text, and beam search still seeks high likelihood. (right) "Beam search reduces the risk of one poor early choice but can still loop."
• "Top-k and top-p are the same." (wrong) One keeps a count and the other a probability mass. (right) "Top-p adapts the pool size; top-k does not."
• "Decoding changes the model." (wrong) The weights stay the same. (right) "Decoding acts on the same model's output distribution."
• "Higher temperature makes the model smarter." (wrong) It only flattens the probabilities. (right) "Temperature trades predictability for variety."

Follow-up questions

What are decoding strategies in LLMs (temperature, top-k, top-p, beam search)?

Decoding strategies are the rules that choose a token from the model's probabilities: greedy decoding takes the top token, beam search tracks several sequences, and temperature, top-k and top-p shape a random draw. The Hugging Face documentation lists top_k, top_p and repetition_penalty among its generation parameters.

Your model repeats phrases and gets stuck in loops on long generations. How do you fix degeneration?

Move away from pure highest-probability decoding toward sampling with top-p, since Holtzman et al. tie degeneration to maximum-likelihood decoding. A repetition penalty is another lever; in Hugging Face, a value of 1.0 means no penalty. Verify on your own prompts, because no single setting is a universal fix.

Why not always pick the most likely word?

Human text has fluctuating per-token probability, while always taking the top token gives unnaturally flat, high probability and dull, looping text. Sampling restores variety.

Does a decoding strategy change the model?

No. The weights are untouched; the strategy only changes how a token is chosen from the same probabilities. Serving costs such as quantization or paged attention are separate from this choice.

What happens if you raise the temperature very high?

The probabilities approach equal chances, so rare tokens get drawn often and the text turns incoherent. In the toy flat case the top token already falls from 0.3046 at T of 0.5 to 0.2082 at 1.5.¹

Sources

1. The Curious Case of Neural Text Degeneration (arXiv 1904.09751) — arXiv (Holtzman et al.), ICLR 2020, 22 Apr 2019
2. Text generation: GenerationConfig documentation (Transformers main branch, read 2026-10-26) — Hugging Face Transformers, 26 Oct 2026
3. Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation — ACL Anthology (Garces Arias et al., COLING 2025), 19 Jan 2025
4. Conformal Nucleus Sampling (Findings of ACL 2023) — ACL Anthology, 1 Jul 2023