In this article
Before you start
This page sits in the deep learning path and leans on probability and statistics.
Why interviewers ask this
The question has no single right answer, so it tests judgment. An interviewer wants to see whether a candidate follows research beyond the dominant recipe, can explain a new idea in plain words, and can weigh evidence without cheering for it.
A weak answer is a slogan: "diffusion will replace today's chat models" or "it is just hype." A strong answer says how text diffusion generates, what evidence exists, and what is still unproven. A Staff-level answer separates what the papers show from what is only plausible, and names the trade-off that decides use: fewer sequential rounds against weaker joint consistency per round.
This question is a prompt for opinion, so it rewards a structure you can say in about a minute: what it is, what supports it, what it costs, and what you would test.
Text diffusion from first principles
- Text diffusion
- Text diffusion is a family of generative methods that produce text by reversing a gradual corruption process, starting from noise or masks and refining the whole sequence over several rounds, rather than emitting tokens strictly left to right.
- Masked diffusion
- Masked diffusion is a kind of text diffusion in which the corruption replaces tokens with a special mask symbol, and a transformer is trained to predict the hidden tokens, so generation fills masks in over repeated rounds.
- Autoregressive model
- An autoregressive model is a language model that gives each token a probability conditioned on all tokens before it, so it generates one token at a time, left to right, reusing its past outputs as input.
In plain words
Picture a crossword. An autoregressive model fills it in from the first square to the last, never going back. A diffusion model starts with a blank grid. It fills in the squares it is surest about, then revisits the rest with those clues in view. The analogy breaks in one place. A real model decides how many squares to fill per round, and it has no eraser unless the method adds one.
In one precise sentence, text diffusion trains a network to undo corruption of a text and then generates by applying that network repeatedly. A token is the unit of text, a word or word piece, that a language model reads and writes.
Step 1: Start from a fully masked sequence
Generation begins with a sequence of the target length in which every position is a mask. One discrete family, Discrete Denoising Diffusion Probabilistic Models (D3PMs), corrupts tokens with a transition matrix, and Austin et al. (2021) show that the choice of that matrix is an important design decision for image and text results. Masking positions is one such choice. The block length is chosen before generation starts, so the output length is fixed in advance, a real limitation next to a model that simply stops when it is done.
Input: a prompt, if any, plus a block of masks. Output: the same block, waiting to be filled. You now know the starting point; the next step is what the network does with it.
Step 2: Predict every masked position at once
A transformer reads the partly masked sequence and outputs a probability for every masked position together. Nie et al. (2025) describe the training as a masking process in the forward direction and a transformer that predicts masked tokens in the reverse direction, optimizing a likelihood lower bound. Because the network sees context on both sides of a mask, it is bidirectional, unlike a left-to-right model.
Input: the partly masked sequence. Output: a distribution over words at each mask. Training is ordinary backpropagation on a cross-entropy-style loss over the masked positions. Inside, the transformer stacks layers built from activation functions and residual connections. Earlier work on discrete diffusion for language modeling includes Lou et al. (2023) and Sahoo et al. (2024), which LLaDA's masked-token approach sits alongside.
Step 3: Unmasking several tokens per round has a cost
The sampler keeps some predictions and leaves the rest masked. If it keeps only one per round, it matches an autoregressive model's step count. If it keeps several per round, it needs fewer rounds, but each kept token was chosen from its own marginal probability without seeing the others kept in the same round.
A toy case shows the problem. Suppose two neighboring masks should read either "New York" or "Los Angeles" (illustrative), each with probability one half. The marginals are half "New" or "Los" for the first slot and half "York" or "Angeles" for the second. Sampling both independently gives a coherent pair with probability 0.5 and a broken pair such as "New Angeles" with probability 0.5.1
This is the parallel-decoding error. Bidirectional attention also means a standard key-value (KV) cache of past keys and values does not apply, so each round recomputes the whole sequence. Wu et al. (2025) state that open diffusion language models' inference speed often lags autoregressive models because of the missing KV cache and the quality loss when decoding several tokens at once.
Step 4: Repeat until no mask is left
The loop repeats: predict, keep some tokens, re-run. When no mask remains, the text is done. The number of rounds is a dial set at sampling time, so quality and round count trade against each other.
Input: a sequence with fewer masks each round. Output: finished text. The authors of LLaDA trained an 8B-parameter model this way with pre-training and then supervised fine-tuning (SFT), meaning further training on prompt and answer pairs, and report it is competitive with LLaMA3 8B in in-context learning (Nie et al., 2025). The same abstract reports a win over generative pre-trained transformer (GPT)-4o, a model in the generative pre-trained transformer (GPT) family, on one reversal poem completion task.
Step-by-step description
- Every position begins as a mask.
- The transformer scores every mask at once.
- More tokens per round means fewer rounds.
- Loop until no mask remains.
- Generation starts from a fully masked sequence.
- A transformer predicts every masked token at once.
- Unmasking several tokens per round chooses them without seeing each other.
- The loop repeats until no mask is left.
Worked numbers: a ten-token toy
Take a ten-token text (illustrative). A left-to-right model needs 10 sequential steps. A sampler that unmasks 1, 2, 5 or 10 tokens per round needs 10, 5, 2 or 1 rounds, which is ten divided by the tokens per round. In training, the same text is corrupted by masking each token independently with probability 0.5, so about 5 of the 10 are masked on average and the network learns to fill roughly half the text from the other half.1 A sampler runs the other way: it starts with all ten masked and unmasks a few per round, so the last row of the table, ten tokens at once, is the fastest and the riskiest.
| Tokens unmasked per round | Rounds for ten tokens (illustrative) |
|---|---|
| 1 | 10 |
| 2 | 5 |
| 5 | 2 |
| 10 | 1 |
Text description of the diagram
- A ten-token text needs 10 rounds at one token per round.
- Unmasking all ten tokens in one round takes 1 round.
These are round counts, not speeds. A diffusion round runs the whole network over the whole sequence, so fewer rounds do not by themselves mean less time; no speed figure is claimed here.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What is your opinion about text diffusion?
I think it is a credible alternative, not yet a replacement. It generates text by starting from masks and refining all positions over several rounds, so it is bidirectional and not tied to left-to-right order. LLaDA, an 8B model, scales comparably to its authors' own autoregressive baselines and is competitive with LLaMA3 8B on in-context learning (Nie et al., 2025); competitive is not better. The trade-off is that unmasking many tokens per round chooses them without seeing each other, which hurts coherence, and bidirectional attention rules out a standard key-value cache. I would test quality at equal compute and fixed round counts, and try it where infilling matters.
What makes it Staff-level: it states a position, ties each claim to evidence, names the mechanism behind the main cost, and says what you would measure next.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "Diffusion makes images, so text is the same." | Explains masks, a transformer predicting them, repeated rounds. | Also separates continuous and discrete variants and why text needs them. |
| Evidence | A hunch or a headline. | Cites a scaled result such as LLaDA at 8B. | Notes "competitive" is not a win, the scaling comparison uses the authors' own baselines, and one task is not general. |
| Trade-offs | None. | Names parallel decoding versus step count. | Explains the joint-consistency error and that fewer rounds are not speed. |
| Communication | Rambling, buzzwords. | Definition, evidence, cost in order. | Adds a position and the test that would change it. |
| Follow-up readiness | Stuck on "why?" | Handles "how is it trained?" | Handles "what would you measure?" with equal-compute and step-count sweeps. |
Three variants of this question
Do you think diffusion will replace autoregressive language models?
Not on current evidence. Nie et al. (2025) report "competitive" results at 8B against LLaMA3 8B, not a clear win, so a serious second option is the fair label, and results at larger scale and equal compute are what to wait for.
How can diffusion work on text when text is discrete?
Two routes exist. Li et al. (2022) add Gaussian noise in word-vector space and denoise it, since text is inherently discrete, and then map the denoised vectors back to words. Austin et al. (2021) instead corrupt the tokens themselves with a transition matrix.
What would you use text diffusion for first?
Tasks that benefit from editing a whole sequence or from control, since the diffusion language model (LM) of Li et al. reported strong results on six fine-grained control tasks (Li et al., 2022). A plain chat product is not a safe first pick without measured evidence.
Common mistakes
- "Text diffusion is image diffusion on text." (wrong) Text is discrete, so noise needs embeddings plus a mapping back to words, or a discrete corruption process. (right) "It adapts diffusion to discrete tokens, often with masking."
- "Diffusion language models are faster." (wrong) Fewer rounds are not less compute: bidirectional attention rules out a standard KV cache, and decoding many tokens at once lowers quality (Wu et al., 2025). (right) "They can need fewer sequential rounds, but each round runs the whole network, so I would measure speed at equal quality."
- "LLaDA beat GPT-4o, so diffusion is better." (wrong) The abstract reports the win on one reversal poem completion task (Nie et al., 2025), which is encouraging, not general. (right) "It beat GPT-4o on one reversal task, which is a promising signal, not a verdict."
- "Autoregression is required for large language model (LLM) ability." (wrong) LLaDA challenges this at 8B. (right) "Its authors show a diffusion model can reach similar in-context learning, with caveats."
Follow-up questions
Is text diffusion better than autoregressive generation?
The sources do not show it is better overall. LLaDA reports competitiveness with LLaMA3 8B and comparable scaling to its authors' own baselines, which is a strong start, not a verdict.
How do you apply diffusion to discrete text?
Either denoise Gaussian noise in embedding space and map the result back to words (Li et al., 2022), or corrupt tokens directly, for example with an absorbing mask state (Austin et al., 2021).
Why might text diffusion be faster, and why might it not be?
It can finish in fewer sequential rounds when several tokens are unmasked per round. Each round, though, re-runs the network on the whole sequence because bidirectional attention rules out a standard KV cache, and decoding many tokens at once degrades quality; Wu et al. (2025) give these as reasons open diffusion language models often lag autoregressive ones in speed. Total cost and quality at equal compute must be measured.
Does diffusion fix the reversal curse?
Not in general. The LLaDA abstract reports it surpassing GPT-4o on a reversal poem completion task (Nie et al., 2025), which is one task, so it is a promising signal, not a fix. More practice questions are in deep learning interview questions.
- Text diffusion generates by refining a masked or noisy sequence over rounds, not strictly left to right.
- LLaDA 8B is competitive with LLaMA3 8B in in-context learning and scales comparably to its authors' own autoregressive baselines.
- Text is discrete, so diffusion needs embeddings plus a mapping back to words, or a discrete corruption such as masking.
- Unmasking several tokens per round cuts rounds but chooses them without seeing each other, and without a standard KV cache fewer rounds do not mean less time.
- A strong opinion states a position, cites evidence, names the cost and says what to measure.
Sources
- Large Language Diffusion Models (LLaDA) — arXiv / NeurIPS 2025 (Nie et al.); accessed 2026-10-26, 14 Feb 2025
- Structured Denoising Diffusion Models in Discrete State-Spaces — arXiv / NeurIPS 2021 (Austin et al.); accessed 2026-10-26, 7 Jul 2021
- Diffusion-LM Improves Controllable Text Generation — arXiv / NeurIPS 2022 (Li et al.); accessed 2026-10-26, 27 May 2022
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding — arXiv (Wu et al.); accessed 2026-10-26, 28 May 2025
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — arXiv (Lou, Meng, Ermon), 25 Oct 2023
- Simple and Effective Masked Diffusion Language Models — arXiv (Sahoo et al.), 11 Jun 2024
