What happens when one GPU of 128 fails at hour 16?

In this article

1. Before you start
2. Why interviewers ask this
3. One failed GPU in a 128-GPU run from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the machine learning operations (MLOps) interview questions learning path and assumes only that you know what training a model means, as covered in machine learning algorithms and how they work.

Why interviewers ask this

The question tests whether you treat failure as normal at scale rather than as a surprise. Candidates report being asked, in interviews for AI infrastructure roles, what becomes of a 128-GPU training run when a single card dies sixteen hours in. The consequence of a weak answer is a design that loses a day of compute to one dead card.

A weak answer says "the job crashes and we restart it." A strong answer explains why one failure stops everyone, what exactly is lost, and what you restore. A Staff-level answer adds the trade-off: saving state often costs time, saving rarely loses work, and the best interval can be estimated from the failure rate and the cost of one save. It also separates a dead GPU from a slow one, which is the straggler problem in the follow-ups below.

One failed GPU in a 128-GPU run from first principles

Checkpoint

A checkpoint is a saved copy of everything a training run needs to continue, namely the model weights, the optimizer state, the position in the data and the random-number state, written to storage so that a crashed run can resume from it instead of starting over.

Mean time between failures

The mean time between failures (MTBF) is the average running time between two failures of a component or system; for a cluster of independent GPUs it is roughly one GPU's MTBF divided by the number of GPUs.

In plain words

Picture 128 rowers in one long boat. They must pull each stroke in unison. If one oar snaps, the crew stops. They swap the rower and restart from the last buoy they marked. The analogy breaks in one place. Rowers remember where they are. A crashed GPU forgets everything in its memory. So the marked buoy must have been written to disk beforehand.

In one precise sentence, synchronous training is training in which all GPUs finish each step together by combining their gradients, so one lost GPU halts them all.

Step 1: A healthy step needs every rank

A rank is one worker process, usually one per GPU, numbered from 0 to 127 in this illustrative scenario. In data parallelism each rank holds a full copy of the model, computes gradients on its own slice of the batch, and then all ranks run an all-reduce so that every copy applies the same update. The all-reduce is the barrier: nobody finishes the step until everybody has contributed.

Input: 128 sets of gradients. Output: one averaged set on every GPU. You now know why one GPU matters to the others: they cannot complete a step without its numbers. A model that is split across GPUs instead of copied (pipeline or tensor parallelism) is even more tightly coupled, as the distributed training design page explains.

Step 2: The failure is detected and the group stops

At hour 16 one GPU dies, from a hardware fault, a driver error or a lost node. Its rank stops sending data, so the other 127 ranks wait inside the all-reduce. The NVIDIA Collective Communications Library (NCCL) has no timeout of its own, so a silent peer can leave the collective hanging. The framework's watchdog ends the wait: in PyTorch, the process-group timeout set in torch.distributed.init_process_group aborts the stuck collective and raises an error (PyTorch distributed documentation, read 2026-10-26). A launcher process on each machine then sees that a worker has failed.

Input: one silent rank. Output: a decision to tear the group down. Elastic training is training that can re-form its group of workers when machines fail or join, so that a job survives churn; it is documented in the PyTorch elastic documentation. The torchrun launcher documentation lists --max-restarts, which caps how many times the worker group is restarted, and the rendezvous range of minimum and maximum nodes. Nothing continues on 127 GPUs unless the job was built to re-form.

Step 3: The checkpoint interval sets how much work is lost

Work lost equals the time since the last completed checkpoint, multiplied by the 128 GPUs (illustrative scenario) that must redo it. If the last checkpoint finished at hour 15.0, the loss is 1 hour on 128 GPUs, which is 128 GPU-hours.¹ On average a failure lands halfway through an interval, so a 2-hour interval loses 128 GPU-hours on average and up to 256 at worst.¹

Saving more often is not free, because each save pauses or slows training. The balance has a closed-form estimate, Young's first-order result (Young, 1974):

Here $T^{\ast }$ is the best checkpoint interval in hours, $C$ is the cost of one save in hours and $M$ is the cluster MTBF in hours. Daly (2006) refines it by subtracting roughly $C$, which matters only when $C$ is not small against $M$. The Young and Daly interval is the checkpoint spacing that this estimate gives, the one that minimizes total waste. With an assumed 50,000 hours per GPU (illustrative), the cluster MTBF is 390.625 hours, and with a 3-minute save the interval is 6.25 hours.¹ Cheaper saves shorten it: a 0.01-hour (36-second) save gives 2.795 hours.¹

You now know the one number that decides the bill: the interval, not the failure itself.

Step 4: The run resumes with its full state

The restart loads the latest complete checkpoint on every rank, rebuilds the 128-GPU group on healthy machines, and continues from the saved step. A correct resume restores more than weights: the optimizer state (for example momentum, which is extra per-weight memory the optimizer keeps), the learning-rate schedule position, the data-loader position and the random seeds. If the data position is lost, the run repeats or skips examples; if the optimizer state is lost, the next updates are wrong in size.

Input: the last checkpoint and 128 healthy GPUs. Output: a run that continues as if nothing happened, minus the lost interval. Check the loss curve after the resume; it should continue from where it was, not jump.

1 · Healthy step2 · Failure3 · Checkpoint4 · ResumeReplay

Step-by-step description

1. All 128 ranks join one all-reduce.
2. One rank fails and the group stops.
3. The last checkpoint sets the loss.
4. State is restored and training resumes.

• All 128 ranks join one all-reduce each step.
• One failed rank stops the group.
• The group restarts from the last checkpoint.
• Training resumes with weights, optimizer state and data position restored.

Worked example

Take 128 GPUs (illustrative scenario), a failure rate of one per 50,000 GPU-hours and a 3-minute checkpoint, all illustrative.¹ If failures are independent, the chance that at least one GPU fails in the first 16 hours is 4.01%, and a 30-day run expects about 1.843 failures.¹ The failure is rare per day and routine per month, which is why recovery must be designed, not improvised.

The overhead column is the save cost divided by the interval, and the lost-work column is half the interval divided by the cluster MTBF; their sum is the waste.¹ Total waste is 20.03% at a 0.25-hour interval, and total waste is 1.6% at 6.25 hours and 1.95% at 12 hours.¹ Saving every 15 minutes wastes 20.03% of the run on saves alone, while saving every 12 hours wastes 1.95%, mostly on redone work.¹ The curve is flat near its minimum, so an interval of 4 hours is within 0.16 percentage points of the best.¹

Text description of the diagram

• Total waste is 20.03% at a 0.25-hour interval.
• Total waste is 1.6% at 6.25 hours and 1.95% at 12 hours.

This estimate ignores restart time and assumes independent failures, so treat it as a starting point, not a measurement. The 50,000 figure is an assumption, but real clusters land in a similar range. The Llama 3 paper (section 3.3.4 and Table 5) reports 466 job interruptions in a 54-day pre-training snapshot on up to 16,384 H100 GPUs, 419 of them unexpected, with about 78% attributed to confirmed hardware issues. Dividing the GPU-hours (54 × 24 × 16,384) by 419 gives about 50,677 GPU-hours per unexpected interruption, and 54 × 24 / 419 is an interruption roughly every 3.1 hours.¹ This is a rough implied figure: not every interruption is a single-GPU failure, and the GPU count varied.

Trade-offs and edge cases

Plain restart, elastic restart and spare nodes all restore from a checkpoint, and they differ in how the group is rebuilt. A plain restart rebuilds the same 128-GPU group and waits until the failed machine is replaced. Elastic restart lets the group shrink or grow within limits, so training continues on fewer machines, but the global batch size changes unless you adjust the per-GPU batch or accumulate gradients. Spare nodes keep idle healthy machines ready to swap in, which shortens the wait at the price of paying for idle GPUs.

Text description of the diagram

• A plain restart rebuilds the same group after the machine is replaced.
• Elastic restart lets the group shrink or grow, and the batch size changes unless adjusted.
• Spare nodes shorten the wait but cost idle GPUs.

Edge cases matter more than the happy path:

• A failure during a save. Write the new checkpoint to a temporary name and rename it only when complete, and keep the previous one until then, otherwise a crash can leave no usable file.
• A slow save. Asynchronous checkpointing copies state to host memory and writes in the background, which lowers the save cost and therefore, by the formula above, the best interval.
• A silent fault. A GPU that returns wrong numbers without crashing is worse than a dead one; watch the loss for sudden spikes, a habit shared with large language model (LLM) observability.
• A hang, not a crash. NCCL has no timeout of its own, so a silent peer can hang the group until the framework watchdog fires; set the process-group timeout and alert on steps that stop advancing.

How to answer it in an interview

The spoken structure: say the job stops, say why, say what is lost, then give the interval trade-off and what you restore.

Common mistakes

• "The job just continues without that GPU." (wrong) Synchronous training needs every rank for each all-reduce. (right) "The group stops and restarts from a checkpoint, on a re-formed group if the job is elastic."
• "We lose all 16 hours." (wrong) Only the work since the last checkpoint is lost. (right) "We lose at most one checkpoint interval on every GPU."
• "Checkpoint as often as possible." (wrong) Saves cost time; at 15 minutes the example wastes 20.03% on saves alone.¹ (right) "I balance save cost against lost work with the Young and Daly estimate."
• "Weights are enough to resume." (wrong) Without optimizer state and data position the run diverges or repeats data. (right) "I save weights, optimizer state, schedule, data position and seeds."

Follow-up questions

How would you design a training system for a model that does not fit on a single device?

Split the model across devices with pipeline or tensor parallelism and shard optimizer state, then make checkpoints sharded so each device writes its own part. The distributed training design page covers the split; failure handling is the same, only more tightly coupled.

How do you solve the straggler problem in synchronous training across thousands of GPUs?

A straggler is a slow GPU, not a dead one, and it delays every step because the all-reduce waits for the slowest rank. Record per-rank step times, flag the outlier, and find the root cause with health checks for thermal or link degradation, then drain the node and replace it at the next checkpoint. To hide stragglers in the meantime, use backup workers: run N + b workers and apply the first N gradients each step, as in Chen et al. (2016).

What if the failure rate doubles?

The cluster MTBF halves, and the interval shrinks by about 1.41 times, from 6.25 to 4.419 hours in the example.¹ Lost work becomes the larger cost, so you save more often or make saves cheaper.

What if the GPU dies while a checkpoint is being written?

Write to a temporary name, verify, then rename, and keep the previous checkpoint until the new one is complete. The run then resumes from the older save and loses one extra interval.

How is a failure different from model drift?

A failure stops the job and is visible at once; drift degrades a deployed model silently over months, as in trained in January, bad in July. Resuming a run exactly is also a reproducibility and governance concern.

Sources

1. torch.distributed.elastic documentation (read 2026-10-26) — PyTorch, 26 Oct 2026
2. torchrun (Elastic Launch) documentation (read 2026-10-26) — PyTorch, 26 Oct 2026
3. Distributed communication package, init_process_group timeout (read 2026-10-26) — PyTorch, 26 Oct 2026
4. A first order approximation to the optimum checkpoint interval — Communications of the ACM (Young), 1 Sep 1974
5. A higher order estimate of the optimum checkpoint interval for restart dumps — Future Generation Computer Systems (Daly), 1 Feb 2006
6. Revisiting Distributed Synchronous SGD (arXiv 1604.00981) — arXiv (Chen et al.), 4 Apr 2016
7. The Llama 3 Herd of Models (arXiv 2407.21783) — arXiv (Meta AI), 31 Jul 2024