How do you estimate model training duration?

In this article

1. Before you start
2. Why interviewers ask this
3. Estimating training time from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the take a model to production path, and the training loop it relies on is covered in machine learning algorithms and how they work.

Why interviewers ask this

Training duration decides cost, schedule and whether a plan is feasible at all, so a system design round on training often asks for it within the first few minutes. The question separates candidates who can reason from first principles from those who quote a number.

A weak answer says "it depends" or "a few days" with no arithmetic. A strong answer names the formula, states the inputs and says what it assumed. A Staff-level answer also says which input is least certain (almost always achieved throughput), runs a short measurement to pin it down and adds the time lost to failures and restarts. The distributed training design page shows where this estimate sits inside a full design.

Estimating training time from first principles

Training compute

Training compute is the total number of floating-point operations a training run performs, from the first step to the last. For a dense transformer it is roughly six times the parameter count times the number of training tokens.

Hardware utilization

Hardware utilization is the fraction of a chip's theoretical peak speed that a real job achieves, after communication, data loading and optimizer work are included. It is closely related to what practitioners call model FLOPs utilization (MFU). It is always below one hundred percent, and it is the least certain input to a time estimate.

In plain words

Think of moving a pile of sand with trucks. The pile is the compute. Each truck's carrying rate is a GPU's throughput. Time is pile size divided by the total rate of all trucks. The analogy breaks here: trucks on a clean road carry their full rate, while GPUs lose time waiting for data and for each other, so the rate you get is lower than the rate on the label.

In one precise sentence, training time equals training compute divided by the number of GPUs times the throughput each GPU actually sustains, plus time lost outside the training steps.

A token is a small chunk of text, usually a word or part of a word, that a language model reads as one unit. A dense transformer is a model in which every parameter is used for every token.

1 · Count compute2 · Measure speed3 · Divide4 · Add overheadReplay

Step-by-step description

1. Six times parameters times tokens in FLOPs.
2. Sustained throughput per GPU, not peak.
3. Compute over GPUs times throughput.
4. Failures, checkpoints and queues.

• Training compute is about six times parameters times tokens.
• Throughput is measured, not read from the datasheet peak.
• Time is compute divided by GPUs times throughput, plus overhead.

Step 1: Count the compute the job needs

Training compute for a dense transformer is about six times parameters times tokens, written $C\approx 6ND$ with $N$ parameters and $D$ tokens. The six comes from two parts. The forward pass costs about two operations per parameter per token. Weng (2026) counts the backward pass as twice the forward pass, because backpropagation runs two matrix multiplications, giving six in total.

The scaling-law paper of Kaplan et al. (2020), summarized in study notes, writes the same quantity as six times N times B times S, with B tokens per update and S updates, so D equals B times S. The six leaves out the attention term, which grows with context length, so the estimate drifts low at very long sequence lengths.

Input: parameter count and token count. Output: total FLOPs. For a model with 175 billion parameters trained on 300 billion tokens, the numbers Narayanan et al. (2021) use for Generative Pre-trained Transformer 3 (generative pre-trained transformer (GPT)-3), that is 3.15e23 FLOPs.¹ You now have the size of the pile; next you need the speed.

Step 2: Measure the throughput the whole system delivers

Throughput is the FLOP/s each GPU sustains across the whole training loop, not the speed of its matrix units alone. The abstract of Narayanan et al. (2021) reports 502 petaFLOP/s on 3,072 GPUs, an achieved per-GPU throughput of 52% of theoretical peak. Dividing 502 petaFLOP/s by 3,072 gives about 163 teraFLOP/s per GPU (a teraFLOP/s is a trillion FLOPs per second), and dividing that by 0.52 implies a peak near 314 teraFLOP/s per GPU.¹ That was a heavily optimized record-scale run on A100 GPUs, so a typical job should expect no better.

This is the common error. A candidate who divides by peak speed gets an answer that is too short. At a round 312 teraFLOP/s peak (illustrative), the same GPT-3-scale run on 1,024 GPUs takes 21.9 days at 52% utilization but 57.1 days at 20%.¹ How you find the real number is in Step 3.

Text description of the diagram

• At 20% utilization the run takes 57.1 days.
• At 52% utilization the same run takes 21.9 days.

Step 3: Divide, then check the result against a known run

Time in seconds is compute divided by the product of GPU count and per-GPU throughput. Take 1,024 GPUs sustaining 140 teraFLOP/s each (illustrative inputs). The GPT-3-scale job needs 3.15e23 FLOPs, so the run takes 2,197,266 seconds, about 25.4 days.¹

Activation recomputation is a memory-saving technique that re-runs part of the forward pass during the backward pass. It adds roughly another two operations per parameter per token, so the count becomes eight times parameters times tokens, and the same job takes 33.9 days.¹ Use six when you do not recompute, eight when you do, and say which.

To get the throughput input without guessing, run a pilot. After warm-up, time about 100 to 200 training steps on the target cluster and read the cluster's tokens per second. Then time equals total tokens divided by tokens per second. Per-GPU delivered FLOP/s equals six times N times tokens per second, divided by the GPU count, and dividing that by the datasheet peak gives utilization. In the example above, 1,024 GPUs at 140 teraFLOP/s for 175 billion parameters imply 136,533 tokens per second; 300 billion tokens at that rate take 2,197,266 seconds, the same 25.4 days, and 140 over the illustrative 312 teraFLOP/s peak is 44.9% utilization.¹

Check the method against a published run. Narayanan et al. (2021, §5.1) work through a trillion-parameter model trained on 450 billion tokens with 3,072 GPUs. At 163 teraFLOP/s each, six gives 62.4 days and eight gives 83.2 days, a little under three months.¹ The abstract's 502 petaFLOP/s is the same throughput in aggregate: 3,072 GPUs times about 163.4 teraFLOP/s.¹

Step 4: Add the time lost outside the training steps

The division gives time spent in training steps. A real run also loses time to checkpoint saves, evaluation passes, failed GPUs, restarts from the last checkpoint and waiting in a queue. These have no clean formula, so state a margin as an assumption. With a 10% overhead (illustrative), the 25.4-day job at 45% utilization becomes 27.9 days.¹ That is the same job as in Step 3, because 45% of the illustrative 312 teraFLOP/s peak is about 140 teraFLOP/s.

Failure risk grows with GPU count and run length, so the margin should too; see what happens when one GPU of 128 fails at hour 16. Queue time depends on how the cluster is shared, which is a scheduling question covered in Kubernetes for machine learning infrastructure. Give the answer as a range with the main assumption, not as one number.

The Staff-level answer

What makes it Staff-level: it names the weakest input, replaces it with a measurement and ends in a range with a stated assumption.

The rubric

Three variants of this question

How long will fine-tuning on my dataset take?

Use the same formula with $D$ equal to the fine-tuning tokens times the number of passes over them. Then measure tokens per second on a short run of your own hardware, because fine-tuning setups vary widely in memory tricks and sequence lengths.

How do you estimate training time when there is no clean FLOP count?

For small neural networks and other non-transformer models, measure instead. Run a few hundred steps, time them, and multiply by total steps. If 300 steps take 120 seconds (illustrative) and the job needs 78,130 steps (2,000,000 samples, batch size 256, ten passes), the estimate is 8.68 hours.¹ Check whether the data loader, not the model, limits the step time. Tree ensembles have no batches in this sense: time a few boosting rounds and multiply by the planned rounds, remembering that cost grows roughly with rows times features.

What if the dataset doubles?

Time roughly doubles, because training compute grows linearly with tokens at a fixed model size. If you also scale the model to match, compute grows faster, as the Chinchilla result in the follow-ups explains.

Common mistakes

• "Time is FLOPs divided by the GPU's peak FLOP/s." (wrong) Even a record-setting run reached 52% of peak. (right) "I divide by measured throughput, which I get from a short pilot run."
• "Double the GPUs and the time halves." (wrong) Near-linear scaling needs fast links and a good parallelism setup, as Narayanan et al. (2021) report for an optimized cluster. (right) "I assume some efficiency loss and say what it is."
• "Training costs two operations per parameter per token, like inference." (wrong) Inference is the forward pass only. (right) "Training adds the backward pass, so it is about six."
• "The token count is a fixed input." (wrong) It is a design choice. (right) "I choose tokens with the compute budget in mind, then estimate time."
• "The formula gives the finish date." (wrong) It ignores failures and queues. (right) "The formula gives the training time; I add overhead and give a range."

Follow-up questions

What if the number of GPUs doubles?

In the ideal case the time halves: the 25.4-day job at 45% utilization falls to 12.7 days on 2,048 GPUs. If scaling efficiency is only 85% (illustrative), it takes 14.9 days.¹ The gap is communication cost between more machines.

How do you account for failures and restarts?

Add a margin for work lost between the last checkpoint and each failure, plus the restart time, and let the margin grow with GPU count and run length. Frequent checkpoints shrink lost work but cost time to write.

How many tokens should you train on?

Tokens are a choice tied to model size. Hoffmann et al. (2022) found that for every doubling of model size, training tokens should also double, and that many large models were significantly undertrained. Their abstract describes a 70-billion-parameter model trained on 4× more data than Gopher, and the paper's Table 1 (Hoffmann et al., 2022) lists 1.4 trillion tokens for Chinchilla, which is 20 tokens per parameter and 5.88e23 FLOPs.¹

Why not just use a bigger batch to finish sooner?

A bigger batch cuts the number of serial steps only up to a point. The critical batch size is the batch size beyond which extra tokens per update add compute without cutting the step count in proportion, as the Kaplan study notes put it, summarizing Kaplan et al. (2020). Past it you pay for more compute and save little time.

Does the same estimate apply to serving a model?

No. Serving is the forward pass for each request, and the questions there are latency and cost per request; see batch, streaming and real-time inference.

Sources

1. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (arXiv 2104.04473) — arXiv (Narayanan et al.), 9 Apr 2021
2. Training Compute-Optimal Large Language Models (arXiv 2203.15556) — arXiv (Hoffmann et al.), 29 Mar 2022
3. Scaling Laws, Carefully — Lilian Weng's blog, 24 Jun 2026
4. Scaling Laws for Neural Language Models (arXiv 2001.08361) — arXiv (Kaplan et al.), 23 Jan 2020
5. Notes on Scaling Laws for Neural Language Models (Kaplan et al.) — University of Waterloo, CS 886 course notes (undated; accessed 2026-10-26), 26 Oct 2026