How do you design an image-generation product end to end?

In this article

1. Before you start
2. Why interviewers ask this
3. An image-generation product from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one design in the generative AI system design guide.

Why interviewers ask this

A weak answer describes a model and forgets the product. Candidates have reported this prompt at large tech companies in senior and principal-level rounds, and it asks for the whole chain: data collection and curation, the architecture, the training objective, and the compute budget.

A weak answer names "a diffusion model" and stops. A strong answer says where the training pictures come from and how they are cleaned, why generation runs in a compressed space, how the prompt steers the result, what one image costs to serve, and how you decide the product is good. A Staff-level answer adds the failure modes: unsafe outputs, copied training images, and a reference object that drifts between generations. The question is the same shape as other end-to-end designs such as design Perplexity or design a voice agent: name the stages, then spend the time on the stage that sets cost or risk.

An image-generation product from first principles

Latent diffusion model

A latent diffusion model is a generative model that runs the diffusion process in the compressed latent space of a pretrained autoencoder, not on raw pixels, which cuts the compute needed to train it and to sample from it while keeping image quality.

Denoising step

A denoising step is one pass of the trained network that removes a little noise from the current image or latent. A finished image needs many such passes in sequence, so the number of steps is a main driver of latency and cost.

Classifier-free guidance

Classifier-free guidance is a sampling method that combines a conditional and an unconditional estimate from one jointly trained model, with no separate classifier, to trade sample diversity for closer adherence to the prompt.

In plain words

Think of a sculptor who starts with a rough block and removes a little stone each pass. After many passes a shape appears. Generation here works the same way, starting from pure noise. The analogy breaks in one place. The sculptor sees the target, while the model only has the prompt and what it learned from training pictures.

A diffusion model is a generative model that learns to undo noise, and it forms an image by applying denoising autoencoders in sequence, as Rombach et al. (2022) describe. In one precise sentence: the product feeds a prompt and noise to a trained network, repeats a denoising step many times, decodes the result to pixels, and checks it before it reaches the user.

Step 1: Training data is collected, cleaned and captioned

The data step decides what the model can draw and what legal and safety risk you carry. Collect pairs of an image and a caption from licensed, partner or public sources, and record the licence of each source. Remove near-duplicates, because Carlini et al. (2023) extracted training images from diffusion models and found that images duplicated in the training set were most at risk.

Filter out unsafe, low-quality and watermarked pictures. Replace weak alt-text captions with richer descriptions where you can, since the caption is the main signal that links words to pixels, and check on a held-out prompt set that it helps. Gathering the pairs is a crawling problem of its own; see how to design a crawler for large language model (LLM) training data.

Input: raw image and text pairs. Output: a clean, deduplicated, captioned set with a licence record. In a worked example below the funnel keeps 400 million of one billion pairs, with every rate illustrative.¹ You now know what the model learns from; the next step is how the image is represented.

Step 2: An autoencoder compresses images into a latent space

Pixel-space diffusion is costly. Rombach et al. (2022) note that training in pixel space often consumes hundreds of GPU days and that inference is expensive because the evaluations are sequential. Their fix is to apply diffusion in the latent space of a powerful pretrained autoencoder, which lets training run on limited computational resources while retaining quality.

Input: an image. Output: a latent. A latent is the much smaller grid of numbers that the autoencoder's encoder produces for the image; the decoder turns a latent back into an image. For an illustrative 512 by 512 colour image compressed to a 64 by 64 grid with 4 channels, the latent holds 48 times fewer numbers.¹ The denoiser works on the small grid and the decoder runs once at the end.

Step 3: Guidance and step count set the cost per image

The denoiser predicts and removes noise, one denoising step at a time, while the prompt steers it. Conditioning is any extra input, such as text, that steers what the network generates. A text encoder, such as a CLIP (Contrastive Language-Image Pre-training) text encoder, turns the prompt into embeddings, and the denoiser reads them through cross-attention while it denoises. Classifier-free guidance, from Ho and Salimans (2022), jointly trains a conditional and an unconditional model and combines their estimates to trade sample quality against diversity. A higher guidance scale follows the prompt more closely and gives less varied images.

Guidance and step count set the cost per image. A guided step needs both estimates, so one image costs about steps times two network evaluations. With 50 steps that is 100 evaluations; at an assumed 20 evaluations per second on one GPU, an image takes 5 seconds, or 720 images per GPU-hour (illustrative).¹ Cut the steps or drop the second estimate and throughput rises sharply: in the worked example, 4 unguided steps give 25 times the throughput of 50 guided ones (illustrative).¹

Step 4: Object consistency trades identity against cost

Text alone cannot pin down a specific object. Three levers can: a zero-shot adapter, per-subject tuning and spatial control. You choose by how many objects you have and how exact the identity must be.

The image-prompt adapter of Ye et al. (2023), called image prompt (IP) adapter, adds an image prompt to a pretrained model through a lightweight adapter of about 22 million parameters. Its key design is decoupled cross-attention: separate cross-attention layers for text and image features. The base model stays frozen, so no per-object training is needed. The cost is identity: the authors note that results match the reference mainly in content and style, less than Textual Inversion or DreamBooth.

Per-subject tuning is the high-fidelity route. DreamBooth (Ruiz et al., 2022) tunes a model on one subject from 3 to 5 images, and low-rank adaptation (LoRA), which trains a small low-rank update instead of every weight, keeps each tune small. It gives the highest identity fidelity, from a short training job per object. Spatial control is the third lever: ControlNet (Zhang et al., 2023) feeds an extra input such as an edge map or a pose to a frozen model. It fixes layout, not identity.

Input: the reference image or images and the text prompt. Output: a picture that follows both.

Text description of the diagram

• An adapter needs no per-object training.
• Per-subject tuning gives the highest identity fidelity.
• Spatial control fixes layout, not identity.

Step 5: Safety, evaluation and serving make a product

A model is not a product until three things exist. First, safety: screen the prompt before generation and the image after it, because either can be unsafe, and mark outputs with a watermark or provenance metadata. Second, evaluation: CLIPScore (Hessel et al., 2021) measures image-text agreement, and Fréchet inception distance (FID) (Heusel et al., 2017) compares feature statistics of generated and real images. Pair them with human preference comparisons on a fixed prompt set. Third, serving: batch requests, cache the text embeddings, and decide the step budget per user tier.

Input: a request. Output: an image that passed the checks, plus a log. Training and serving are separate lanes of the pipeline. Training runs once: data, autoencoder, then denoiser training. The serving path runs the prompt check, the text encoder and adapter, the denoiser loop, the decoder that turns the latent into pixels, the output check and then the user. The adapter conditions the denoiser loop. Pipelines like this look like a GenAI platform with a gateway in front, where limits, routing and safety checks usually sit.

1 · Data2 · Latent space3 · Guided denoiser4 · Adapter5 · Safety and servingReplay

Step-by-step description

1. Clean, deduplicated, captioned pairs.
2. An autoencoder compresses images.
3. Steps and guidance set the cost.
4. Text and a reference condition the denoiser.
5. Decode, screen and return the image.

• Training and serving are separate lanes of the pipeline.
• The adapter conditions the denoiser loop.
• The decoder that turns the latent into pixels runs before the output check.

Worked example

The numbers below come from a short script and every input is an assumption, not a measurement.¹ Start with the data funnel: one billion raw pairs, 20% removed as near-duplicates, then half of the rest passing the quality and safety filters, leaves 400 million training pairs.

Then the cost. The scenarios assume 20 network evaluations per second per GPU (illustrative). Scenario A uses 50 steps with guidance, so 100 evaluations. Scenario B halves the steps to 25. Scenario C assumes a model distilled to need 4 steps with no second guidance estimate, a what-if you would have to build and test. Few-step distillation trains a student network to match a many-step teacher, as in latent consistency models and adversarial diffusion distillation, and guidance distillation (Meng et al., 2022) removes the second evaluation. The 25-fold gap between scenarios A and C is illustrative.¹

Text description of the diagram

• Illustrative throughput is 720 images per GPU-hour at 50 guided steps.
• Illustrative throughput is 1,440 at 25 guided steps and 18,000 at 4 steps without a second pass.

The lesson is the shape, not the values. Latency and cost scale with steps times evaluations per step, so you state those two levers first and say you would measure the real values on your hardware.

Trade-offs and edge cases

Quality against cost is the main trade. A higher guidance scale trades diversity for fidelity to the prompt (Ho and Salimans, 2022). Step count mainly affects sample quality and convergence: denoising diffusion implicit models (DDIM) report sampling 10× to 50× faster in wall-clock time, at similar quality, than the original denoising diffusion probabilistic model (DDPM) sampler (Song et al., 2020). Fewer steps lower the bill and may lower quality on your model, so you measure both.

What loss trains the denoiser?

The denoiser is trained to predict the noise that was added to a training latent, and the loss is the mean-squared error between the true noise and the prediction, the simplified objective of Ho et al. (2020). Newer systems train with rectified flow, a flow-matching objective in which the network learns the direction that moves noise toward data along a near-straight path, and they use transformer denoisers. Esser et al. (2024) scale rectified-flow transformers for high-resolution images, and Peebles and Xie (2022) build diffusion models on transformers. State the loss in one sentence, then say what conditioning you add.

How does the product reach high resolution?

Cost grows with the number of latent positions. For the same 8-fold downsampling, doubling the side from 512 to 1,024 pixels gives four times the latent positions (illustrative).¹ Products therefore generate at a base resolution and add detail with a latent upsampler or a cascade, which is a second model that refines a smaller image. They also train on several aspect ratios so the model does not crop everything to squares. Name the choice and the extra cost of the second stage.

Diffusion or autoregressive?

Autoregressive image generation turns an image into a sequence of discrete tokens and decodes them one at a time with a transformer, as Parti does (Yu et al., 2022). Compare on three points. Sequential decoding needs one network pass per token, while diffusion runs a fixed number of steps over the whole grid. Text conditioning is natural for token models as a prefix, and diffusion adds it through cross-attention and guidance. Inpainting suits diffusion, which denoises a masked region while holding the rest fixed. For an image-only product, commit to latent diffusion; practice differs between labs, so present it as a trade-off, not a fact.

Edge cases: a reference object can drift when the adapter is weak, so raise the adapter weight, tune per subject or add ControlNet; copied training images appear when duplicates stay in the data; and an unsafe prompt can slip through a text-only check, which is why the output image is screened as well.

How to answer it in an interview

The spoken structure: data, latent model, cost per image, the consistency trade-off, then safety and evaluation.

Common mistakes

• "Diffusion runs on pixels." (wrong) That is the expensive version. (right) "Latent diffusion runs in an autoencoder's latent space, which saves compute."
• "Higher guidance is always better." (wrong) It trades diversity for prompt match. (right) "I tune the guidance scale and accept less variety."
• "I always fine-tune the full base model per object." (wrong) That is costly and hard to reuse. (right) "For many new objects I use an adapter; for exact identity I tune per subject, with a low-rank update to keep each tune small."
• "Safety is one filter on the prompt." (wrong) An image can be unsafe from a harmless prompt. (right) "I screen the prompt and the output."

Follow-up questions

How do you design an adapter for object consistency from a reference image?

Freeze the base model, encode the reference image into embeddings, and feed them through their own cross-attention layers beside the text ones, as IP-Adapter does; train only the adapter and expose a weight. That gives content and style, not exact identity. For exact identity tune per subject with DreamBooth (Ruiz et al., 2022), optionally with a low-rank update, and add ControlNet when layout must hold. Choose by how many objects you have and how exact the match must be.

Diffusion or autoregressive: which do you pick?

Pick latent diffusion: it runs a fixed step budget that distillation can shrink, instead of one pass per image token, and it comes with guidance, adapters and natural inpainting. Autoregressive models such as Parti are the alternative, and I would revisit them if the product had to share one token model with text. Say that the choice is a trade-off and commit.

How do you cut the cost per image?

Name the biggest lever first: few-step distillation, such as latent consistency models or adversarial diffusion distillation, so the model needs about 4 steps (illustrative), plus guidance distillation to drop the second evaluation. Then use faster samplers such as DDIM, reduced-precision inference, batching and cached text embeddings. Measure quality after each change, because every cut can lower it.

How do you evaluate an image-generation system?

Use CLIPScore, which measures image-text agreement, and FID, which compares generated and real image statistics, for fast regression checks. Use human preference comparisons on a fixed prompt set for the decision, and track safety failures separately.

What does a higher guidance scale do?

It makes the image follow the prompt more closely and makes the set of images less varied, the quality and diversity trade-off described by Ho and Salimans.

Sources

1. High-Resolution Image Synthesis with Latent Diffusion Models (arXiv 2112.10752) — arXiv (Rombach et al.), 13 Apr 2022
2. Classifier-Free Diffusion Guidance (arXiv 2207.12598) — arXiv (Ho and Salimans), 26 Jul 2022
3. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (arXiv 2308.06721) — arXiv (Ye et al.), 13 Aug 2023
4. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (arXiv 2208.12242) — arXiv (Ruiz et al.), 25 Aug 2022
5. Adding Conditional Control to Text-to-Image Diffusion Models (arXiv 2302.05543) — arXiv (Zhang et al.), 10 Feb 2023
6. Denoising Diffusion Probabilistic Models (arXiv 2006.11239) — arXiv (Ho et al.), 19 Jun 2020
7. Denoising Diffusion Implicit Models (arXiv 2010.02502) — arXiv (Song et al.), 6 Oct 2020
8. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv 2403.03206) — arXiv (Esser et al.), 5 Mar 2024
9. Scalable Diffusion Models with Transformers (arXiv 2212.09748) — arXiv (Peebles and Xie), 19 Dec 2022
10. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (arXiv 2206.10789) — arXiv (Yu et al.), 22 Jun 2022
11. Latent Consistency Models (arXiv 2310.04378) — arXiv (Luo et al.), 6 Oct 2023
12. Adversarial Diffusion Distillation (arXiv 2311.17042) — arXiv (Sauer et al.), 28 Nov 2023
13. CLIPScore: A Reference-free Evaluation Metric for Image Captioning (arXiv 2104.08718) — arXiv (Hessel et al.), 18 Apr 2021
14. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (arXiv 1706.08500) — arXiv (Heusel et al.), 26 Jun 2017
15. Extracting Training Data from Diffusion Models (arXiv 2301.13188) — arXiv (Carlini et al.), 30 Jan 2023
16. On Distillation of Guided Diffusion Models (arXiv 2210.03142) — arXiv (Meng et al.), 6 Oct 2022