In this article
Before you start
This page sits in the transformer architecture learning path and uses ideas from how neural networks work.
Why interviewers ask this
A wrong picture of Sora costs a candidate the whole follow-up chain, because every later question (cost, consistency, scale) hangs on knowing what the tokens are. Interviewers use the question to see whether you can carry a transformer idea from text to another kind of data.
A question about training a Sora-class world model on a very large video corpus has been reported by candidates in an AI-lab system design round. It comes from candidate reports, not a public source, so treat it as a sign of the topic and not as a claim about what any company asks.
A weak answer says "it is a generative pre-trained transformer (GPT) for video." A strong answer names the three parts: a compression step, spacetime patches as tokens, and a diffusion transformer that denoises them. A Staff-level answer adds the cost: attention over all patches is quadratic, so compression ratio and patch size decide what is affordable.
How Sora works from first principles
- Spacetime patch
- A spacetime patch is a small block of a compressed video that spans a few positions in height, width and time, and that is flattened into one vector so a transformer can treat it as a single token, the way a language model treats a word piece.
- Diffusion transformer
- A diffusion transformer (DiT) is a diffusion model whose denoising network is a transformer instead of a convolutional network, so it reads a sequence of noisy tokens plus conditioning, such as a text prompt, and predicts cleaner tokens.
- Video latent
- A video latent is a compact grid of numbers, produced by a compression network, that stands for a video with far fewer positions in time and space than the original pixels, and that a decoder can turn back into a watchable video.
In plain words
Picture an old television showing static. Now imagine a process that sharpens the static, a little at a time, into a picture that matches a written description. Generation in Sora is like that: start from noise, remove noise in many passes, steer each pass with the text prompt.
The analogy breaks in one place. Sharpening here is done by a learned transformer on small blocks of a compressed video, not on the pixels you see, and every block is adjusted in each pass together with all the others.
In one precise sentence, Sora is a diffusion model whose denoiser is a transformer working on spacetime patches of a latent video. It is trained on videos and images of variable durations, resolutions and aspect ratios.
A diffusion model is a generator that learns to turn random noise into data by removing noise step by step. A decoder is the network that maps the generated latent back to pixels.
As of October 2026, the only architecture OpenAI has published is the February 2024 report on the first Sora. OpenAI's Sora 2 announcement (30 September 2025) describes a video and audio model with synchronised dialogue and sound effects. The sources used here give no architecture details for it, so the steps below describe the published design.
Step 1: Compress the video into a latent
Sora first shrinks video before any transformer sees it. OpenAI's report describes a network that takes raw video as input and outputs a latent representation compressed in time and in space. Sora is trained on, and later generates in, that compressed space.
The diagram shows all four steps in order; this section covers the first.
Step-by-step description
- Video becomes a smaller latent.
- The latent becomes spacetime patches.
- A transformer denoises all patches together.
- A decoder maps the latent to pixels.
- A compression network turns video into a latent.
- The latent is cut into spacetime patches that act as transformer tokens.
- A transformer denoises all patches together with attention.
- A decoder maps the latent back to pixels.
Input: raw video. Output: a video latent, a small grid of numbers. A matching decoder is trained to map latents back to pixel space.
The compression is the reason the rest is affordable. A transformer over raw pixels would face millions of positions, and the next steps put numbers on that. The report does not state the compression factors, so the figures below are labelled illustrative.
You now know where the work happens: not on pixels, but on a latent. Next, that latent has to become a sequence a transformer can read.
Step 2: Cut the latent into spacetime patches
A transformer needs a sequence of tokens. Sora gets one by slicing the latent into spacetime patches. The report says the patches are extracted from a compressed input video and act as transformer tokens: where language models have text tokens, Sora has visual patches.
The same scheme covers images, because an image is a video with a single frame. That is how one model trains on both.
The number of tokens follows from the sizes. With T frames, height H, width W, temporal compression , spatial compression , patch side p and temporal patch size (how many latent frames one patch spans):
Take illustrative values: a 5-second clip at 24 frames per second (120 frames), 480 by 640 pixels, temporal compression of four, spatial compression of eight, 2 by 2 patches and , so each patch covers one latent frame. The latent holds 30 frames of 60 by 80, which gives 30 × 30 × 40 = 36,000 tokens.1 Without any compression, the same clip cut into 2 by 2 pixel patches would give 9,216,000 tokens, 256 times more.1
Step 3: Denoise all patches together with attention
The core of Sora is a transformer that sees every patch of the clip at once. Training adds noise to the clean latent patches, and the model learns to predict the clean patches from the noisy ones plus conditioning such as the text prompt, as OpenAI describes. At generation time, the output size is set by arranging randomly initialised patches into a grid of the target size, then denoising them over many passes.
The attention is global in the traditional form of diffusion transformers. The Sora team describes spacetime patches talking with all of the other ones, and Kang et al. (2024), who build a model that follows Sora's architecture, apply self-attention to the entire spatio-temporal sequence without separating space and time. OpenAI has not published whether Sora itself factorises attention. Either way, each pass updates the whole patch grid, so the denoising is not an autoregressive process that writes frames one after another.
How much does attention cost per clip?
Attention over spacetime patches costs quadratic compute in the number of tokens. Every token attends to every token, so n tokens need n × n pairs.1 That is the topic of why attention is quadratic; kernels such as FlashAttention cut memory traffic but still compute every pair.
| Clip (illustrative) | Tokens | Attention pairs |
|---|---|---|
| 5 seconds | 36,000 | 1,296,000,000 |
| 10 seconds | 72,000 | 5,184,000,000 |
| 20 seconds | 144,000 | 20,736,000,000 |
| 5 seconds, no compression | 9,216,000 | 84,934,656,000,000 |
Text description of the diagram
- Under illustrative assumptions a 5 second clip has 36,000 tokens.
- Doubling the clip length doubles the tokens but quadruples the attention pairs.
You now know the cost. Multi-head attention runs several such attention maps side by side, so the compute grows with the number of heads.
Step 4: Decode the latent back to pixels
After the last denoising pass, the result is still a latent, not a video. A decoder trained alongside the compression network maps the generated latents back to pixel space.
Input: a denoised latent grid. Output: frames you can watch, with the frame count that the compression network reduced in time restored; in the illustrative example of Step 2, the decoder would turn 30 latent frames back into 120 frames.1 For text-to-video generation only the decoder is needed; the compression network matters for training, where real videos must be turned into latents. If the model is given an existing image or video to extend or animate, that input goes through the compression network too.
The report says its largest model, Sora, can generate a minute of high fidelity video.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How does Sora work?
Sora is a diffusion transformer working in a compressed latent space. A compression network shrinks video in time and space, the latent is cut into spacetime patches that act as transformer tokens, and the transformer is trained to turn noisy patches into clean ones given the text prompt. To generate, I start from random patches in a grid of the target size, denoise them with attention across the whole clip, and a decoder maps the latent back to pixels. The trade-off is cost: attention grows with the square of the patch count, so compression ratio and patch size decide how long and sharp a clip is affordable.
What makes it Staff-level is the last sentence: it ties the architecture to a cost that limits it, and it does not claim numbers OpenAI has not published.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "A language model for video." | Diffusion model with a transformer on latent patches. | Also separates training (compress, add noise) from generation (noise to latent to decoder). |
| Depth | Says "it uses attention." | Explains patches as tokens and the compression step. | Derives token count from clip size, compression and patch size. |
| Trade-offs | None given. | Notes that longer video costs more. | Shows quadratic attention cost and why compression ratio matters. |
| Honesty | States physics is learned. | Says results look physical. | Says whether video models learn physical rules is still an open question. |
Three variants of this question
Is Sora a large language model, or "GPT for video"?
No. Sora is a diffusion model whose denoiser is a transformer, and its tokens are visual patches, not text. A large language model (LLM) predicts the next text token, and GPT stands for generative pre-trained transformer, a family of next-token text models.
Does Sora generate a video frame by frame, or pixel by pixel?
No. Sora generates all spacetime patches of the clip together in a compressed latent space, then decodes to pixels. Frame-by-frame autoregressive video models are a different family.
What changes if the clip is twice as long?
The token count doubles and the attention pairs quadruple (Step 3, illustrative numbers). Doubling both resolution sides also multiplies the tokens by four, so memory and compute, not the idea, set the limit.
Common mistakes
- "Sora predicts the next frame, like a language model predicts the next word." (wrong) Sora denoises many patches together. (right) "Sora starts from noise and denoises spacetime patches jointly, then decodes."
- "Sora works on pixels." (wrong) It works in a compressed latent space. (right) "A compression network produces a latent, the transformer works on its patches, and a decoder returns pixels."
- "Sora has learned the laws of physics." (wrong) That is unproven. (right) "It produces physically plausible clips, but whether such models learn rules is open."
- "The token count is a known Sora figure." (wrong) Published sources give no patch sizes or compression factors. (right) "With assumed ratios, tokens equal frames over temporal compression times the latent grid over patch size squared."
Follow-up questions
How do you handle temporal consistency in generative video models?
Attention across all spacetime patches gives each position global context, and the Sora team says that is how properties like object permanence can fall out. Models that generate frame by frame or shot by shot need extra conditioning to carry the same state forward.
What is a World-Action Foundation Model, and how does it differ from a standard autoregressive language model?
The sources used here do not define the term, so say what you mean by it. The contrast to draw is that an autoregressive language model predicts the next text token from the previous ones, while Sora's published design denoises a whole grid of spacetime patches in a compressed latent space.
Does Sora 2 work the same way?
Not shown. OpenAI's announcement (30 September 2025) presents Sora 2 as a video and audio model with synchronised dialogue and sound effects, and its architecture is not described in the sources used here. The same page says the Sora product is no longer available as of 26 April 2026, so the 2024 report remains the published design reference.
Does Sora really learn physics?
Not shown. Kang et al. (2024) call it an open question whether video generation can discover physical rules merely by watching videos, as Sora does. Say the clips look plausible, and that plausibility is not proof of a rule.
How do you mitigate exposure bias when training generative world models for long-horizon simulation?
Exposure bias is the mismatch between training on true past frames and generating from the model's own earlier outputs, where errors compound. A strong answer names mitigations such as scheduled sampling, which Bengio et al. (2015) introduced to move training gradually from true past tokens to the model's own predictions, or denoising a whole window together instead of one step at a time. Present these as general techniques, not as Sora facts.
How would you train a Sora-style model on a very large video corpus?
Reported by candidates as an interview question. A strong answer covers shard-based streaming data loading, precomputing latents if the compression network is frozen so it is not rerun every epoch, sharding model and optimiser state across graphics processing unit (GPU) memory, and frequent checkpoints. Keep clip and shot boundaries in the data format, so training samples do not silently splice two unrelated shots and continuity across shots can be checked on held-out clips. More practice questions are in the LLM interview questions.
- Sora is a text-conditional diffusion model with a transformer denoiser, not a text-token language model.
- Video is compressed to a latent, then cut into spacetime patches that act as tokens.
- All patches are denoised together with global attention, then a decoder returns pixels.
- Attention cost is quadratic in tokens, so doubling clip length quadruples the pairs.
- Treat "Sora learns physics" and any exact Sora compression figure as unproven.
- The published architecture is the 2024 report on the first Sora; Sora 2's design is not published.
Sources
- Video generation models as world simulators — OpenAI (Brooks, Peebles et al.), 15 Feb 2024
- OpenAI Sora 2 team: how generative video will unlock creativity and world models — Sequoia Capital (Training Data podcast), 4 Nov 2025
- Sora 2 is here — OpenAI, 30 Sep 2025
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (arXiv 1506.03099) — arXiv (Bengio et al.), 9 Jun 2015
- How Far is Video Generation from World Model: A Physical Law Perspective (arXiv 2411.02385) — arXiv (Kang et al.), 4 Nov 2024
