Data parallelism vs model parallelism

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A wrong layout wastes real money on a large language model (LLM) run: a layout that does not fit stops on the first step, and one that fits but moves too many bytes leaves expensive GPUs waiting. A candidate who says "data parallelism is faster and model parallelism is for big models" has the right instinct and no mechanism. Interviewers want to hear what is copied, what is split, what crosses the network and which limit you hit first.

They then push on a number: how much memory does a 70-billion-parameter model need, and which layout fits it. A weak answer treats the two as rival options. A strong answer treats them as layers of one design, chosen in order. This page is part of fine-tuning and LLM training and leads into the end-to-end exercise of designing distributed training.

What each one is

Data parallelism

Data parallelism is a way of training on many GPUs that keeps a full copy of the model on each GPU, feeds each copy a different part of the batch, and averages the gradients across copies before every weight update.

Model parallelism

Model parallelism is a way of training or running a model that is too large for one GPU by splitting the model itself, its layers or the matrices inside them, across several GPUs so that each GPU holds only a part.

Data parallelism in plain words

Picture several cooks, each with a full copy of the recipe book, each cooking from a different crate of ingredients. After every dish they compare notes and agree on one correction to the book, so all the books stay identical. That is data parallelism: every GPU holds all the weights, works on its own slice of the batch, and the gradients are averaged.

In PyTorch, each process holds its own copy of the model. The gradients are averaged with an asynchronous all-reduce as the backward pass runs. The replicas stay identical because they start from the same state and see the same averaged gradients (PyTorch Distributed Data Parallel (DDP) notes, version 2.14, as of October 2026). The analogy breaks here: a real cook can carry a book of any size, but a GPU cannot.

Model parallelism in plain words

Now the recipe book is too thick for any one cook, so you tear it apart. In tensor parallelism you split each page, so every cook holds a slice of every recipe and they combine partial results at each step. Pipeline parallelism is the other split: you divide the book by chapters, so the first cook does the early steps and hands the half-finished dish down the line.

Tensor parallelism splits the matrices inside each layer; pipeline parallelism assigns groups of consecutive layers to different GPUs (Shoeybi et al., 2019; Huang et al., 2018). Model parallelism trades memory for communication, because the GPUs must now exchange layer outputs. The analogy breaks here: cooks hand over a dish for free, but GPUs pay time for every byte they exchange.

Why one full copy per GPU stops fitting

On an 80 GB GPU such as the H100 (as of October 2026), a full copy per GPU stops fitting at about five billion parameters, because training needs far more memory than the weights alone, and the threshold is card memory divided by 16 bytes. With mixed-precision Adam, each parameter takes 2 bytes for the weight, 2 for its gradient and 12 for the optimizer state, a total of 16 bytes (Rajbhandari et al., 2019, section 3.1). The H100 SXM lists 80 GB of memory (NVIDIA, as of October 2026), and 80 GB divided by 16 bytes is 5.0 billion parameters.

NVIDIA lists 141 GB on the H200 (NVIDIA H200 page, as of October 2026), which moves the line to 8.8 billion.¹ A model of 7 billion parameters therefore needs 112 GB of states, 13 billion needs 208 GB and 70 billion needs 1,120 GB, before any activations. Plain data parallelism cannot train any of them on one 80 GB card per replica. What you now know: the question is rarely "which is faster", it is "does a replica fit".

Text description of the diagram

• Data parallelism copies the model to every GPU and splits the batch.
• Tensor parallelism splits each layer, and pipeline parallelism splits the layers into stages.

Worked example

Take a 70-billion-parameter model on 80 GB GPUs, with 16 bytes per parameter, so its model states total 1,120 GB. A full copy per GPU needs 1,120 GB and fails.¹ Sharded data parallelism, as in the Zero Redundancy Optimizer (ZeRO) and the PyTorch Fully Sharded Data Parallel (FSDP) method, still gives each GPU a different slice of the batch, but it splits the weights, gradients and optimizer states across the data-parallel GPUs instead of copying them (Rajbhandari et al., 2019; Zhao et al., 2023).

Over 64 GPUs that is 17.5 GB each. Tensor parallelism of degree 8 combined with 4 pipeline stages uses 32 GPUs per replica, which leaves 35 GB each. Both layouts fit the model states, but memory still differs in the activations, the intermediate values kept for the backward pass. Sharding leaves each GPU the activations of its whole sequence through all layers. Tensor parallelism with sequence parallelism, which also splits the layer-norm and dropout regions along the sequence, divides them by the tensor degree (Korthikanti et al., 2022, section 4.2).

For the illustrative shape below at 4,096 tokens and one sequence per GPU, that is about 91.3 GB under sharding even when the attention scores are recomputed, so sharding needs full activation recomputation (5.4 GB of layer inputs); tensor 8 by pipeline 4 holds about 11.4 GB at the busiest stage under the one-forward-one-backward schedule.¹ Traffic is the other half of the choice.

Traffic is the second number, and it must be compared for the same model in the same unit. Standard data parallelism moves about twice the parameter count in elements per GPU per step, written 2Ψ with Ψ the parameter count (Rajbhandari et al., 2019, section 7.1). The exact ring all-reduce factor is 2(n minus 1) over n, which follows from the 2(n minus 1) transfers per element in the NVIDIA Collective Communications Library (NCCL) tests notes, as of October 2026.

The gradients of 70 billion parameters in 16-bit form are 140 GB, so plain data parallelism over 64 GPUs would send 275.6 GB per GPU per step, if it fitted, which it does not. Sharded data parallelism over those 64 GPUs moves 1.5 times that, about 3(n minus 1) over n times 140 GB, or 413.4 GB per GPU per step (Rajbhandari et al., 2019, section 7). That traffic is paced by the links between servers.¹

Tensor parallelism pays per layer: a transformer layer needs two all-reduces in the forward pass and two in the backward pass (Shoeybi et al., 2019). For an illustrative 70B shape (layer width 8,192, a sequence of 4,096 tokens, 80 layers), tensor parallelism of degree 8 on its own would need 140 GB of states per GPU and does not fit. In the layout that fits, tensor 8 by pipeline 4, each GPU holds 20 layers and sends 9.40 GB per micro-batch.¹

Now put both layouts on the same global batch of 64 sequences. Tensor 8 by pipeline 4 on 64 GPUs is 2 replicas of 32 GPUs, so each replica runs 32 micro-batches of one sequence. Each GPU then sends about 300.6 GB per step over the fast links inside a server, and the 2 replicas add a gradient all-reduce of 4.38 GB per GPU between servers.¹ Tensor-parallel traffic grows with the tokens each GPU processes, so at 64 sequences it stays 300.6 GB however a replica's 32 sequences are cut into micro-batches. Sharding sends 413.4 GB per GPU per step, paced by the links between servers.

The totals are the same order, so the choice is not "less data". It is where the bytes travel: almost all over in-server links for tensor by pipeline, over inter-server links for sharding, plus pipeline hand-offs and the bubble.

Text description of the diagram

• A full copy of a 70-billion-parameter model's states is 1,120 GB at 16 bytes per parameter.
• Sharded over 64 GPUs it is 17.5 GB, and with tensor 8 times pipeline 4 it is 35 GB per GPU.

Trade-offs and edge cases

Tensor parallelism stays inside a server

Its all-reduces happen in every layer, so they need the fastest links. The authors of the Megatron follow-up paper write that tensor parallelism works well up to about 20 billion parameters on servers of eight 80 GB GPUs, and breaks down beyond that because the all-reduces must cross slower inter-server links, with small matrix multiplications also lowering GPU utilization (Narayanan et al., 2021). Rack-scale systems move the boundary: NVIDIA describes its 72-GPU Blackwell rack as one NVLink domain with 130 TB/s of GPU communication (NVIDIA rack-scale page, as of October 2026), so on that hardware "inside a server" becomes "inside a rack".

Pipeline parallelism has a bubble

A pipeline bubble is the idle time that GPUs spend waiting at the start and end of each step while the stages fill and drain. A micro-batch is a slice of the batch that is fed through the pipeline one after another. GPipe gives the bubble as proportional to (K minus 1) over (M plus K minus 1), where K is the number of stages and M the number of micro-batches.

With equal stage times the idle share is exactly that fraction. GPipe found the overhead negligible once M is at least 4 times K (Huang et al., 2018). For 4 stages the idle share is 75.0% with 1 micro-batch, 42.9% with 4, 27.3% with 8, 15.8% with 16 and 8.6% with 32.¹

Text description of the diagram

• For 4 stages the idle share is 75.0% with 1 micro-batch.
• The idle share is 27.3% with 8 micro-batches and 8.6% with 32.

Combined and sharded splits

Large runs combine tensor, pipeline and data parallelism. The follow-up paper trains a 1-trillion-parameter model at 502 petaFLOP/s (502 quadrillion floating-point operations per second) on 3,072 GPUs, which the authors describe as 52% of theoretical peak (Narayanan et al., 2021). Data parallelism has a ceiling of its own: the number of devices cannot exceed the batch size, and beyond a point the per-GPU batch becomes too small (the same paper, section 1).

The generative pre-trained transformer (GPT) model GPT-3 trained at a batch of 1,536, so data parallelism alone reaches only 1,536 GPUs (section 5.4.3). Sharding changes the picture. FSDP reports performance comparable to plain data parallelism while supporting significantly larger models (Zhao et al., 2023), which is why it is often tried before splitting layers. Its cost is extra communication to gather weights when they are needed.

Context and expert parallelism

Newer splits address other limits than weight memory. Context or sequence parallelism divides a long input across devices; Ring Attention does this for attention and overlaps the transfers with compute (Liu et al., 2023). Expert parallelism, used for mixture-of-experts models, places different experts on different GPUs. In an interview, name the limit each axis relieves: weight memory, activation memory, compute or sequence length.

The order of the choices

The order of choices follows the figure below. For the GPU facts behind these limits, see GPU fundamentals, high-bandwidth memory and the memory of optimizer states; for the single-machine case, parallel processing.

1 · Replica fits2 · Does not fit3 · Too slow, batch cap, activations4 · Beyond a serverReplay

Step-by-step description

1. Use plain data parallelism.
2. Shard the states across data-parallel GPUs.
3. Split layers with tensor parallelism in a server.
4. Add pipeline stages across servers.

• Plain data parallelism is the first choice while a full replica fits on one GPU.
• Tensor parallelism stays inside one server and pipeline parallelism spans servers.

How to answer it in an interview

The spoken structure is: what is copied or split, the memory limit, then the order of choices.

Common mistakes

Treating them as rivals. (wrong) "Pick data or model parallelism." (right) "Large runs combine tensor, pipeline and data parallelism, chosen in order." It is wrong because each one relieves a different limit.

Counting only the weights. (wrong) "A 7B model is 14 GB, so it fits on one GPU for training." (right) "Training needs about 16 bytes per parameter, so 7B needs 112 GB before activations."¹ It is wrong because gradients and optimizer state are stored too.

Putting tensor parallelism across servers. (wrong) "Split every layer over 64 GPUs." (right) "Keep tensor parallelism inside a server, where the links are fastest." It is wrong because its all-reduces happen in every layer.

Ignoring the bubble. (wrong) "Four pipeline stages give four times the speed." (right) "With few micro-batches most of the time is idle: 75.0% with one."¹ It is wrong because stages wait for each other to fill and drain.

Follow-up questions

What does FSDP shard, and when does it gather?

It shards the weights, gradients and optimizer state across the data-parallel GPUs, and gathers a layer's weights just before that layer runs, then frees them. Each GPU holds about 1 over N of the states, 17.5 GB instead of 1,120 GB for a 70B model over 64 GPUs, at 1.5 times the traffic of plain data parallelism.¹ It is data parallelism for the batch and sharding for memory. (A question collected from real interview prompts.)

How do you combine data, tensor and pipeline parallelism?

Data parallelism copies the model and splits the batch; tensor parallelism splits each layer and stays on the fastest links; pipeline parallelism splits the layers into stages and leaves a bubble (Huang et al., 2018). The GPU count is the product of the three degrees (tensor 8 × pipeline 4 × data 2 = 64 here), and the layers of one replica take 8 × 4 = 32 GPUs. Tensor parallelism takes the innermost, fastest in-server links, pipeline stages span servers, and data parallelism replicates the whole grid and splits the batch across the replicas (Narayanan et al., 2021). (A question collected from real interview prompts.)

Where does expert parallelism fit next to pipeline and tensor?

Tensor and pipeline parallelism split the matrices and the layers of a dense model. Expert parallelism is the extra axis for mixture-of-experts models: different experts live on different GPUs, so the tokens travel to the experts that handle them. Say which axis relieves which limit: memory, compute or sequence length. (A question collected from real interview prompts.)

How would you design training for a model that does not fit on one accelerator?

Compute 16 bytes per parameter, which is 1,120 GB for 70B, then divide by the GPUs that share the states. Shard over data-parallel GPUs first; if the traffic is too slow, the batch cannot grow or activations do not fit, add tensor parallelism of degree 8 and pipeline stages. Worked answers at the 10-billion to 1-trillion scale are in designing distributed training, and the adapter route that avoids all of this is LLM fine-tuning. (A question collected from real interview prompts, and a phrase people search for.)

What if the card has 141 GB instead of 80 GB?

The threshold is card memory divided by 16 bytes. A 141 GB H200 holds a full replica's states up to 8.8 billion parameters, so a 7B model (112 GB) trains with plain data parallelism there if its activations fit in the remaining 29 GB, and not on an 80 GB card.¹ A 70B model still needs 1,120 GB, so it still needs sharding or a model split. (A what-if variant of the opening question.)

Sources

1. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — arXiv (Shoeybi et al.), 17 Sep 2019
2. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — arXiv (Narayanan et al.), 9 Apr 2021
3. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — arXiv (Huang et al.), 16 Nov 2018
4. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — arXiv (Rajbhandari et al.), 4 Oct 2019
5. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel — arXiv (Zhao et al.), 21 Apr 2023
6. Distributed Data Parallel (PyTorch notes, version 2.14) — PyTorch (date = read date), 24 Oct 2026
7. NCCL tests: performance notes, AllReduce bus bandwidth — NVIDIA (date = read date), 24 Oct 2026
8. NVIDIA H200 Tensor Core GPU — NVIDIA (date = read date), 24 Oct 2026
9. NVIDIA GB200 NVL72 — NVIDIA (date = read date), 24 Oct 2026
10. Ring Attention with Blockwise Transformers for Near-Infinite Context — arXiv (Liu et al.), 3 Oct 2023
11. NVIDIA H100 Tensor Core GPU — NVIDIA (date = read date), 24 Oct 2026
12. Reducing Activation Recomputation in Large Transformer Models — arXiv (Korthikanti et al.), 10 May 2022