How do you design a GPU cluster scheduler and control plane?

In this article

1. Before you start
2. Why interviewers ask this
3. Design a GPU cluster scheduler and control plane from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one prompt in the distributed systems interview round. The sizing follows the method of back-of-the-envelope estimation.

Why interviewers ask this

A GPU scheduler question tests whether a candidate can separate three jobs that are easy to blur: deciding who may run, deciding where it runs, and keeping the machines honest. Candidates report prompts of this kind in design rounds at AI infrastructure companies, for example a scheduler for 10,000 GPUs running 1,000 mixed training jobs, a control plane with host monitoring and job dispatch, or a GPU credit system; these are reports by candidates, not public statements by any company.

A weak answer draws one box called "scheduler" and a database. A strong answer names the state store, the queue, the placement step and the node agent, and says what each does when something fails. A Staff-level answer adds the cost of a wrong choice: a free GPU that no job can use, or a half-started job holding GPUs forever.

Design a GPU cluster scheduler and control plane from first principles

GPU cluster scheduler

A GPU cluster scheduler is a component that decides which job runs on which GPUs of which machines and when it starts, using the jobs' requests, the free capacity and the rules of priority and fairness.

Control plane

A control plane is the set of services that store the desired and the actual state of a cluster, run the scheduler and other decision loops, and send commands to the machines that do the work.

Gang scheduling

Gang scheduling is a placement rule that starts all the processes of a job together or none of them, because a distributed training job cannot make progress until every process has its GPU.

In plain words

Think of an airport. Passengers are jobs, gates are GPUs, and the control tower is the control plane. The tower keeps one board of what is planned and what is real. A queue manager decides who may board next. Ground staff at each gate report back. The analogy breaks in one place: a training job is a tour group that cannot leave until every seat is free at once. So you cannot seat the group one by one.

In one precise sentence, the control plane records what you want, the scheduler picks a placement that satisfies it, and node agents make the machines match.

Step 1: A replicated state store and node agents hold the truth

The state store is a database that holds every job, every node and every assignment, behind an application programming interface (API) that is the only way to change it. Several copies of the API server and the store run so that one failure does not stop the cluster; the database interview questions cover replication and consensus.

A node agent is a small program on each machine that reports its GPUs, starts the processes it is told to start and sends a heartbeat. A heartbeat is a small periodic message that says "this node is alive and these GPUs are healthy". A reconciliation loop is a loop that compares desired state with actual state and acts on the difference. Input: job and node records. Output: one shared view. Every later step reads and writes this view, never talking to a node directly.

Step 2: Gang scheduling starts a whole job or none of it

A distributed training job runs one process per GPU, and every process must be present before any can make progress. A scheduler that places processes one at a time can place half of two jobs and strand both. In the illustrative case below, two jobs need eight GPUs each and twelve are free; placing six for each uses all twelve and no job can start, whereas gang scheduling admits one job and leaves four GPUs free.¹

The fix is to treat the job as one unit with a minimum size. A gang job asks for all its GPUs or none. The scheduler either finds room for the whole gang in one decision, or leaves the job pending and holds nothing. The Kueue documentation has an all-or-nothing scheduling page for this rule. You now know the unit the scheduler places: the gang, not the process.

Step 3: A quota queue admits jobs before any placement

A quota is a limit on how many GPUs a team may hold. A queue holds submitted jobs and admits one only when its team is inside its quota, so admission is a separate decision from placement. The Kueue documentation names the same layer with concepts such as cluster queues, cohorts, preemption and fair sharing. Preemption is stopping a lower-priority running job to free GPUs for a higher-priority one.

For fair share, give each team a weight. With weights 5, 3 and 2 on 10,000 GPUs, the floors are 5,000, 3,000 and 2,000 GPUs; if the third team is idle, its 2,000 are lent in a 5:3 ratio, so the first team may use 6,250.¹

A credit system is the same idea in money form: reserve credits at admission, commit the real usage, and refund the rest. A job of 8 GPUs for 24 hours at one credit per GPU-hour reserves 192 credits from a balance of 1,000; if it runs 10 of the 24 hours it commits 80 and refunds 112.¹ Make each step idempotent, as the payment system design does.

Step 4: Placement filters the nodes, then scores them

Placement is a two-stage search. The scheduler first filters out nodes that cannot run the job, such as the wrong GPU type, too few free GPUs or an unhealthy status, then scores the feasible nodes to pick the best one (Cloud Native Computing Foundation (CNCF), 2021). The Kubernetes GPU documentation describes configuring GPUs as a resource that nodes advertise, which is the filter's input.

The score decides fragmentation. Fragmentation is the state where many GPUs are free but scattered in pieces too small for the jobs waiting. Pack small jobs onto already-busy nodes so whole nodes stay free for large ones, and put the members of a multi-node gang close together on the fastest links. The scheduler writes its choice to the state store; it does not start anything itself. Scheduler replicas run with a single active leader chosen by leader election, and optimistic writes with version checks stop a stale replica from taking the same GPU, as the concurrency interview questions explain.

Step 5: Dispatch, heartbeats and recovery keep the cluster honest

The node agent watches the state store for assignments to its node, starts the containers and reports status. It also reports GPU health signals, such as memory errors or a GPU that has fallen off the bus, so the control plane can drain a node whose GPU is failing even while its heartbeats continue. The control plane marks a node suspect when heartbeats stop for a few intervals, stops placing on it and requeues its jobs. A long training job should save checkpoints, so a requeued gang restarts from the last one instead of from zero. Heartbeat cost is small: 1,250 nodes beating every 10 seconds is 125 messages per second, and every 30 seconds is about 42.¹

The tricky case is a node that is slow, not dead. Waiting too short evicts healthy nodes; waiting too long wastes a gang's GPUs, because one dead process stalls every other process. Pick the timeout from how long a restart costs. Rolling out new agents safely is the subject of deployment strategies, and job metrics belong in a metrics and logging pipeline.

1 · State2 · Gang3 · Quota4 · Place5 · RunReplay

Step-by-step description

1. One store holds the truth.
2. A job asks for all its GPUs or none.
3. A queue admits jobs inside quota.
4. Filter nodes, then score them.
5. Agents run the job and send heartbeats.

• A state store and node agents hold the truth about the cluster.
• A gang job asks for all its GPUs or none.
• A quota queue admits the job before placement.
• Placement filters nodes then scores them, and node agents run the job and send heartbeats.

Worked example

Take the prompt "10,000 GPUs, 1,000 heterogeneous jobs". With eight GPUs per node (an assumption, illustrative) the cluster has 1,250 nodes; the jobs average ten GPUs each if they fill it.¹ The script draws 862 jobs of 1, 2, 4, 8, 16 or 64 GPUs until 90% of the cluster is requested, which is 8,988 GPUs and 10.43 GPUs per job on average (illustrative mix).¹

Jobs of 8 GPUs or more take whole nodes; smaller jobs share a node. First-fit (the first node with room) and best-fit (the node with the least room left that still fits) both place all 862 jobs and leave 126 whole nodes free. A spreading rule that always picks the emptiest node rejects 55 jobs and leaves no whole node, although 2,492 GPUs are free.¹

Text description of the diagram

• First-fit and best-fit leave 126 whole nodes free.
• A spreading rule leaves no whole node although 2,492 GPUs are free.

Trade-offs and edge cases

The central trade is utilization against fairness and latency. Fixed floors waste idle GPUs; borrowing raises utilization but needs preemption, and preemption throws away work unless jobs checkpoint. Reported practice differs on how much to preempt, so state the policy you choose and why.

Heterogeneous hardware adds a filter: each job lists the GPU types it accepts, and a pool of scarce types needs its own queue. Placement scoring needs topology, because a job spanning nodes runs at the speed of its slowest link. Packing plus all-or-nothing admission can starve large gangs: freed GPUs keep going to small jobs, so a 64-GPU job (illustrative) at the head of the queue waits indefinitely. The usual fix is to reserve nodes for the oldest large gang as they drain.

Smaller jobs are backfilled onto those nodes only if their declared runtime ends before the reservation starts. Backfill depends on runtime estimates and loses work when they are wrong. Fractional GPU sharing is an extra mechanism that this design treats as out of scope; schedule whole GPUs first. For a text-to-video service, add inference jobs with deadlines, which sit in a high-priority queue so they do not wait behind long training runs.

Text description of the diagram

• The scheduler decides which node runs a job.
• The quota queue decides whether it may start.
• The node agent decides whether its host is healthy.

The scheduler decides which node runs a job, the quota queue decides whether it may start, and the node agent decides whether its host is healthy. The default Kubernetes scheduler places one pod at a time, so queueing and gang rules live in a layer above it, as in Kueue. Rate-limiting submissions at the API is a separate problem covered in the distributed rate limiter design.

How to answer it in an interview

The spoken structure: say the two planes, then the queue and gang rule, then placement, then failure, then the trade-off.

Common mistakes

• "The scheduler is the control plane." (wrong) It is one component. (right) "The control plane also holds state, the API, loops and node agents."
• "I place the job's processes one by one." (wrong) Two big jobs can each hold half the GPUs and both wait forever. (right) "I admit and place the whole gang or nothing."
• "Scheduling is queueing." (wrong) Placement and admission are separate decisions. (right) "A quota queue admits; the scheduler places."
• "I pack small jobs and never reserve for big ones." (wrong) Large gangs starve. (right) "I reserve draining nodes for the oldest large gang and backfill only jobs that finish first."
• "Spread jobs evenly for balance." (wrong) It leaves no whole node for large jobs. (right) "I pack small jobs and keep whole nodes free."

Follow-up questions

How would you design a scheduler for 10K GPUs running 1000 heterogeneous training jobs?

Use the pipeline above: quota queue, gang admission, a filter on GPU type and free count, and a packing score. State the sizing, such as 1,250 nodes at eight GPUs (illustrative), and what you do when the large jobs cannot find whole nodes.

How do you design the control plane for a compute cluster with host monitoring and job dispatch?

Node agents send heartbeats and GPU health to a replicated state store; loops mark nodes suspect and requeue; agents watch for assignments and start them. Dispatch is a write to the store, never a direct call to a node.

How do you design the GPU job scheduler for a text-to-video generation service?

Treat each generation request as a short job with a deadline, queue it by priority, and keep a reserved pool so interactive requests do not wait behind training. Batch similar requests onto one GPU where the model allows.

How do you implement a GPU credit management system?

Keep a ledger with reserve, commit and refund entries, each with an idempotency key, and check the balance at admission. Reconcile reserved credits against actual GPU time when the job ends.

How do you design a GPU-aware pod scheduler?

Model Node, Gpu, Job (or gang) and Scheduler classes, where the scheduler has filter, score and bind steps. Keep a node table with free GPUs by type, filter on request, score by best fit, and bind with a version check. Test it with a gang that does not fit, which must stay pending and hold nothing.

What should happen when a node dies in the middle of a job?

The missed heartbeats mark the node suspect, the whole gang is stopped, and the job is requeued at its old priority to resume from its latest checkpoint.

Sources

1. Advanced Kubernetes pod to node scheduling — Cloud Native Computing Foundation (CNCF), 27 Jul 2021
2. Schedule GPUs — Kubernetes documentation, 20 Sep 2024
3. Kueue overview — Kubernetes SIGs (Kueue documentation, v0.20 as shown in its navigation, read 2026-10-26), 26 Oct 2026