What are diffusion models? How they work, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. Diffusion models from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Diffusion models and their close relatives underlie major text-to-image systems, such as the rectified-flow transformers of Esser et al. (2024), so a candidate who cannot explain them cannot discuss cost, quality or failure in those products. The question comes in several forms: explain how diffusion generates an image, compare DDPM (denoising diffusion probabilistic models) with DDIM (denoising diffusion implicit models), or say what the guidance weight does.

A weak answer says "it starts from noise and cleans it up". A strong answer says what the network is trained to predict, why one training step needs no loop, and what sampling costs in network evaluations. The three-part answer to practise is: the noising maths, the training target, the sampling cost. This page builds each part from zero.

Diffusion models from first principles

Diffusion model

A diffusion model is a generative model that learns to reverse a gradual noising process, so that it can turn random noise into a new sample by removing noise over many small steps.

Forward process

The forward process is the fixed, learning-free procedure that turns a real data point into noise by adding a small amount of Gaussian noise at each of T steps.

Noise schedule

A noise schedule is the list of numbers, one per step, that sets how much noise the forward process adds at that step.

In plain words

Picture a photograph left in the rain. Each minute more drops blur it. After an hour it is a grey smear. Now suppose you had watched thousands of photographs go through that rain. You learned what one more minute does at each stage. You could then take a grey smear and undo the rain one minute at a time. The analogy breaks in one place: the rain here is random static, added by a formula, and the learner is only ever asked one small question, "what static was added?".

In one precise sentence: a diffusion model fixes a forward process that adds Gaussian noise, trains a network to predict that noise at every step, and generates new data by running the network backwards from pure noise. The method was made practical for images by Ho, Jain and Abbeel (2020).

Step 1: Choose a noise schedule

The schedule decides how fast a clean image becomes noise. The original paper sets T = 1,000 steps and a linear schedule in which the noise variance, called beta, rises from 0.0001 at step 1 to 0.02 at step 1,000 (Ho et al., 2020). Each step multiplies the image by the square root of one minus beta and adds Gaussian noise with variance beta.

The input is the pair of end values and T; the output is a list with one beta per step. Nothing is learned here. A useful derived number is alpha-bar at step t, the product of (1 minus beta) over steps 1 to t. It measures how much of the original image's variance survives, and it is the only quantity the next step needs. What you now know: the schedule is a fixed table, and alpha-bar is its running summary.

Step 2: Add noise in one jump

Any noise level has a closed form, because every step adds independent Gaussian noise. The noisy image equals the square root of alpha-bar times the clean image, plus the square root of one minus alpha-bar times a fresh noise sample (Ho et al., 2020, Eq. 4). So training never loops through 500 steps to reach step 500; it jumps straight there.

The numbers show what the jump produces. With the linear schedule, alpha-bar is 0.897 at step 100, 0.524 at step 250, 0.0786 at step 500 and 0.00004 at step 1,000, so only 7.86% of the image's variance remains halfway and essentially none at the end.¹ Why 7.86% is still enough to learn from: the network is never asked to rebuild the picture from that smear. It is asked only for the noise that was mixed in, it is told the step number t, and it is trained at every noise level, so at step 500 it sees a noisy input whose noise it can mostly separate. What you now know: any noise level is one formula away.

Step 3: Train a network to predict the noise

Training picks a real image, a random step t and a noise sample, builds the noisy image with the Step 2 formula, and asks the network to output the noise that was used. The loss is the squared difference between the true noise and the prediction, averaged over many such draws (Ho et al., 2020, Eq. 14). It is a plain regression, trained with gradient descent, not a game between two networks and not cross-entropy.

A worked case: for one pixel with true noise 0.5 and a prediction of 0.2, the loss is 0.09.¹ The network also receives t, so it knows how noisy its input is. Predicting noise is equivalent to predicting the clean image, because the Step 2 formula can be rearranged to recover one from the other. What you now know: the whole model is one noise predictor shared across all steps.

Step 4: Sample by removing noise step by step

To generate, start from pure Gaussian noise and repeat for t from 1,000 down to 1. Predict the noise, subtract a scaled amount of it, rescale the result by 1 over the square root of that step's alpha (one minus beta), and add a small fresh random term except at the last step (Ho et al., 2020, Algorithm 2). After 1,000 passes the result is a new sample.

The cost is the catch. Training needed one network call per example, but sampling needs one call per step, so a 1,000-step sample costs 1,000 forward passes of the network.¹ This is why early diffusion models were slow, and why most of the later research attacks the step count. The random term means two runs from the same start give different images. What you now know: generation is a loop of small corrections, and the loop length sets the cost.

Step 5: Steer the sample with guidance

Classifier-free guidance is a sampling method that steers a conditional diffusion model, such as a text-to-image model, by mixing its prediction with and without the condition. Ho and Salimans train one network that sometimes sees the condition and sometimes a blank, then combine the two predictions as (1 + w) times the conditional one minus w times the unconditional one (Ho and Salimans, 2022).

With illustrative predictions of 0.9 (conditional) and 0.3 (unconditional), a weight w of 0 gives 0.9, w = 1 gives 1.5 and w = 3 gives 2.7.¹ A larger w pushes the sample toward the condition, which raises fidelity to the prompt and lowers variety. The paper reports the best Fréchet inception distance (FID), a lower-is-better image-quality score, with small guidance and the best Inception score (a higher-is-better score of how clear and varied generated images look) with w of 4 or more. What you now know: guidance is one extra weight, paid for with a second forward pass.

1 · Schedule2 · Noise jump3 · Train4 · Sample5 · GuideReplay

Step-by-step description

1. A fixed table sets the noise at each step.
2. Any noise level is one formula away.
3. The network predicts the noise that was added.
4. Remove predicted noise one step at a time.
5. Mix two predictions to follow the prompt.

• The original schedule uses 1,000 steps.
• Training jumps to any noise level with one formula.
• Sampling needs one network call per step.

Worked example

The schedule of the original paper gives the table below. Every row is computed from T = 1,000 and the beta range of Step 1, not typed by hand.¹

Text description of the diagram

• At step 500 only 7.86% of the signal variance is left.
• Only 0.004% of the signal variance remains at step 1,000.

Take one illustrative pixel with clean value 0.8 and a noise draw of 0.5. At step 250 the noisy value is 0.7239 x 0.8 + 0.6899 x 0.5 = 0.9241. If the network predicts the noise exactly, the clean value comes back as (0.9241 - 0.6899 x 0.5) / 0.7239 = 0.8.¹ A real network never predicts perfectly, so its error at high noise levels is what blurs or distorts a generated image.

The core of the training step is short:

t = random_step(1, T)                    # uniform over 1..T
eps = random_normal(like=x0)             # fresh noise sample
x_t = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps
loss = ((eps - network(x_t, t)) ** 2).mean()

Trade-offs and edge cases

DDPM versus DDIM. DDIM is a sampler that reuses a trained DDPM network but jumps to an earlier step with an update that predicts the clean image from the current noisy one. The DDIM paper defines a non-Markovian forward (noising) process, one whose steps are not each tied to only the previous step, that gives the same noise level at each step given the clean image as DDPM, which is why the trained DDPM network still applies. Because training depends only on those noise levels, a sampler can visit a short subsequence of the steps without retraining.

The sampling update uses only the current noisy image, and with its randomness setting, eta, at 0 it is deterministic, which keeps few-step samples good: at 10 steps on a small-image benchmark from the Canadian Institute For Advanced Research (CIFAR), called CIFAR-10, FID is 13.36 with eta = 0 against 41.07 with eta = 1 (Song et al., 2022, Table 1). The same paper reports samples 10 to 50 times faster in wall-clock time with the same training procedure (Song, Meng and Ermon, 2020, revised 2022). With 50 steps instead of 1,000, the network is called 20 times less often.¹

Text description of the diagram

• DDPM samples in 1,000 steps in the original paper.
• DDIM reuses the DDPM training procedure and samples 10 to 50 times faster.

Guidance has a price. Classifier-free guidance needs the network twice per step, once with the condition and once without, so guided sampling costs two forward passes per step.¹ Reported practice differs on the best weight: the original paper found small guidance best for FID and strong guidance best for Inception score, so the weight depends on the metric you value.

Latent diffusion. A latent space is the compressed internal representation of images that an autoencoder (a network that compresses an image into a small code and rebuilds it) learns, and it is much smaller than the pixel grid. Running diffusion on pixels is expensive; Rombach et al. (2021, revised 2022) instead run it in the latent space of a pretrained autoencoder, and add cross-attention layers (the mechanism in multi-head attention) so text can condition the model. A downsampling factor of 8 on a 512 by 512 image leaves 4,096 spatial positions instead of 262,144, a 64-fold cut (illustrative arithmetic).¹

The field since 2022. Four developments change the practical answer, each from a paper opened on 2026-10-24. Consistency models (Song et al., 2023) are built for one-step generation, with multistep sampling to trade compute for quality, and report a one-step FID of 3.55 on CIFAR-10. Flow matching (Lipman et al., 2022) trains by regressing onto vector fields along fixed paths from noise to data, includes diffusion paths as a case, and samples with off-the-shelf solvers for ordinary differential equations (ODEs); Esser et al. (2024) scale rectified flow, a straight-path variant, in a transformer for text-to-image.

Diffusion Transformers (Peebles and Xie, 2022) replace the U-Net backbone (the image-to-image convolutional network earlier diffusion models used) with a transformer over latent patches, and their largest model reaches FID 2.27 on class-conditional ImageNet 256x256. Lin et al. (2023) show that common schedules do not enforce zero signal-to-noise ratio at the last step, so training never sees the pure noise that sampling starts from, and they propose rescaling the schedule.

The schedule matters. The linear schedule has a known weakness. Nichol and Dhariwal (2021) say it destroys information faster than necessary, that the end of the forward process is too noisy to add much to sample quality, and that a model trained on it loses little in image quality (FID) when up to 20% of the reverse process is skipped. They propose a cosine schedule that adds noise more slowly and changes little near the extremes (opened 2026-10-24 in the ar5iv rendering of the paper).

A schedule that leaves too much signal at the final step makes the pure-noise starting point wrong. The linear schedule above leaves 0.004% of variance at step 1,000 (signal coefficient 0.0064).¹ Lin et al. (2023), Table 1 still count it among the schedules without zero terminal signal-to-noise ratio, but its residual is far smaller than Stable Diffusion's signal coefficient of 0.068, so the flaw is harder to notice.

How to answer it in an interview

The spoken structure is: the forward noising and its closed form, the noise-prediction loss, then the sampling cost and how to cut it.

Common mistakes

"A diffusion model is a generative adversarial network (GAN) with a different loss." (wrong) A GAN trains two networks against each other; a diffusion model trains one network with a plain regression loss. (right) "It is trained to predict the noise, with no discriminator."

"Sampling is as cheap as training." (wrong) Training takes one network call per example, sampling one per step. (right) "Training jumps to any noise level; sampling must walk back through the steps."

"The network outputs the finished image in one go." (wrong) It outputs a noise estimate, and the image emerges only after the whole loop. (right) "It predicts the noise at each step; the image is the result of removing it repeatedly."

"DDIM needs a new model." (wrong) DDIM changes only the sampler. (right) "DDIM reuses the DDPM network and takes fewer, deterministic steps."

Follow-up questions

What is the difference between DDPM and DDIM?

DDPM samples with the full chain of small random steps, 1,000 in the original paper. DDIM defines a non-Markovian forward process with the same noise level at each step given the clean image, so the same trained network applies, and its sampling update predicts the clean image and jumps to an earlier step. DDIM skips steps, and with its randomness setting eta at 0 the update is deterministic, which keeps quality when it does; it is reported as 10 to 50 times faster (Song et al., 2022).

How would you use a diffusion model for image denoising?

Treat the noisy photograph as a point partway along the noising path. Estimate its noise level, pick the step t that matches, and rescale the photograph by the square root of alpha-bar at t so it looks like the noisy image the network saw in training at that step (written x_t in the papers). Then run only the reverse steps from t down to 1 instead of starting at pure noise, which fits mild, roughly Gaussian noise. Starting from a higher t to edit a photo is a different use, because it repaints content.

Why do diffusion models need so many steps?

Each reverse step assumes the noise is small enough that undoing it is a simple Gaussian correction. Large jumps break that assumption, so the original method uses 1,000 small steps. DDIM skips steps, and its deterministic update keeps quality when it does. Few-step generation goes further: consistency models are trained to map a noisy point straight to a clean sample, giving one-step or few-step sampling (Song et al., 2023), and flow-matching models learn paths that ODE solvers can follow in fewer evaluations (Lipman et al., 2022).

What happens if you double the guidance weight?

The guided prediction moves further from the unconditional one: with the illustrative scores above, w = 3 gives 2.7 and w = 6 gives 4.5, so the output follows the prompt more closely and loses variety.¹ Past a point image quality falls.

How are diffusion models different from GANs?

A GAN (generative adversarial network) trains a generator against a discriminator in one adversarial game and produces a sample in a single pass. A diffusion model trains one network with a plain squared-error regression on noise, with no discriminator, and pays for that with many sampling passes, which is the cost the few-step methods above attack.

Why does latent diffusion run faster?

It runs the same loop on a much smaller representation, so each network call touches far fewer positions; the latent diffusion paper reports significantly lower compute than pixel-space models. For related questions see AI engineer interview questions.

Sources

1. Denoising Diffusion Probabilistic Models (arXiv 2006.11239 v2) — arXiv (Ho, Jain, Abbeel), 16 Dec 2020
2. Denoising Diffusion Implicit Models (arXiv 2010.02502 v4) — arXiv (Song, Meng, Ermon), 5 Oct 2022
3. Classifier-Free Diffusion Guidance (arXiv 2207.12598) — arXiv (Ho, Salimans), 26 Jul 2022
4. High-Resolution Image Synthesis with Latent Diffusion Models (arXiv 2112.10752 v2) — arXiv (Rombach et al.), 13 Apr 2022
5. Improved Denoising Diffusion Probabilistic Models (arXiv 2102.09672) — arXiv (Nichol, Dhariwal), 18 Feb 2021
6. Consistency Models (arXiv 2303.01469) — arXiv (Song et al.), 2 Mar 2023
7. Flow Matching for Generative Modeling (arXiv 2210.02747) — arXiv (Lipman et al.), 6 Oct 2022
8. Scalable Diffusion Models with Transformers (arXiv 2212.09748) — arXiv (Peebles, Xie), 19 Dec 2022
9. Common Diffusion Noise Schedules and Sample Steps are Flawed (arXiv 2305.08891) — arXiv (Lin et al.), 15 May 2023
10. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv 2403.03206) — arXiv (Esser et al.), 5 Mar 2024