In this article
Before you start
This page is one part of the machine learning operations (MLOps) learning path. The machine learning algorithms overview covers the models that these clusters train and serve.
Why interviewers ask this
Candidates report that platform and MLOps loops at senior and staff level spend real time on Kubernetes, because a team that trains and serves models depends on it daily. A bad cluster answer costs the whole team: stuck jobs, idle graphics processing units (GPUs) and services that fall over under load.
The question separates people who have used kubectl from people who understand why the system behaves as it does. A weak answer lists objects: pods, deployments, services. A strong answer explains the reconcile loop, then shows how GPUs, custom job types and autoscaling plug into it. A Staff-level answer adds failure behavior: what happens when an event repeats, when two workers touch one object, when no GPU is free, or when a metric arrives late.
Kubernetes for ML infrastructure from first principles
- Controller
- A controller is a control loop that watches the state of a Kubernetes cluster and makes or requests changes to move the current state closer to the desired state, which is the spec field of the objects it tracks. It keeps running and never reaches a final "done".
- Operator
- An operator is a software extension to Kubernetes that uses a custom resource to manage an application and its components. It acts as the controller for that resource and encodes domain knowledge such as upgrades, backups and scaling, so users only describe the state they want.
- Extended resource
- An extended resource is a countable resource that Kubernetes does not build in, such as a GPU, which a node advertises by name so that the scheduler can count it like central processing unit (CPU) or memory.
In plain words
Think of a thermostat. You set a target temperature. The thermostat keeps checking the room and turns the heater on or off. You never tell it "heat now". The analogy breaks in one place: a cluster has thousands of targets at once, and each has its own loop.
Kubernetes works the same way. You write down what you want: "run this training job" or "keep this model server running". Loops compare what exists with what you wrote and close the gap.
In one precise sentence: Kubernetes for ML infrastructure is a set of reconcile loops, extended with GPU support, custom job types and autoscalers, that run training and serving.
Step 1: A controller reconciles current state toward desired state
A controller is a loop that moves the current state of the cluster toward the desired state. The Kubernetes documentation says objects carry a spec field that represents the desired state, and that each controller tries to move the current state closer to it.
The same page warns that, potentially, the cluster never reaches a stable state. So a controller must not depend on seeing every event. This is the level-based idea. An edge-triggered (edge-based) handler reacts to each change it is told about. A level-based (level-triggered) reconcile compares the whole current state with the desired state and acts on the difference. If an event is missed, the next pass still finds the gap. If an event repeats, the second pass finds nothing left to do, provided the actions are safe to repeat.
Input: the spec and the observed state. Output: the changes that close the gap. You now know why a controller's code should be written to be safe to run again.
Step 2: An operator turns a custom resource into ML work
An operator is a controller for a custom resource, and it is how ML jobs get first-class support. A custom resource definition (CRD) is a Kubernetes object that adds a new kind of object, such as a training job or a model server, to the cluster's application programming interface (API). The operator page says operators are clients of the Kubernetes API that act as controllers for a custom resource, and that they follow the control loop principle.
The Cloud Native Computing Foundation (CNCF) operator white paper describes the pattern as managing resources with domain knowledge and declarative state, so the user only describes the desired state. For ML, a user writes one small object, "train this model on four workers", and the operator creates the pods, restarts failed ones and cleans up.
Input: a custom resource. Output: ordinary pods and services. An operator is not a separate system; it is a controller you add.
Step 3: A device plugin makes GPUs schedulable
A device plugin is a component that advertises special hardware on a node to Kubernetes. The device plugin documentation describes plugins that register with the kubelet. The kubelet is the agent that runs on each machine and starts its pods. The node then reports the hardware as a vendor-named extended resource such as nvidia.com/gpu, and the scheduler counts it like CPU or memory. Kubernetes has no native concept of a GPU, so something must announce them (ScaleOps, secondary source).
The Schedule GPUs page says a pod asks for GPUs under limits, in whole units; a fraction of a GPU cannot be requested. KubeNatives (secondary source) adds that the NVIDIA plugin runs as a DaemonSet. A DaemonSet is a workload that keeps one copy of a pod on every matching node, here on each GPU node. A pod waits in the Pending state until a node has enough free GPUs.
Placing all workers of a distributed job together, or none at all, is called gang scheduling. Without it, some workers take GPUs and wait for the rest, and the job and its neighbors deadlock; see distributed training design.
Step 4: An autoscaler adds replicas from queue length
A horizontal pod autoscaler (HPA) is a controller that changes the number of copies of a workload to match a metric. When configured with a queue-length metric, the autoscaler adds replicas when queue length per pod exceeds its target. The documentation states that with the autoscaling/v2 API you can give several metrics, and that the autoscaler takes the maximum scale recommended for each metric, provided this is not larger than the maximum you configured.
Resource metrics such as CPU come from metrics.k8s.io. Custom metrics, such as model latency or GPU use, need an adapter and the API aggregation layer; the same page says custom metrics have been stable since Kubernetes v1.23. External metrics, which carry a signal from outside the cluster such as a queue's backlog, also come from an adapter, through the external.metrics.k8s.io API described in the HPA documentation.
Input: metrics. Output: a replica count. Whether a queue, a stream or live requests drive it depends on the serving style; see batch, streaming and real-time inference.
Step-by-step description
- A controller closes the gap to the spec.
- A custom resource describes an ML job.
- A plugin makes GPUs countable.
- Queue length sets replicas.
- A controller moves the current state toward the declared state.
- An operator is a controller for a custom resource.
- A device plugin advertises GPUs as an extended resource that the scheduler counts.
- When configured with a queue-length metric, the autoscaler adds replicas when queue length per pod exceeds its target.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A serving deployment starts with four replicas. Each replica has a queue of 40 requests against a target of 20, and GPU use of 90 percent against a target of 60 percent (all illustrative). The autoscaler uses the documented rule desired = ceil(current × current metric ÷ target metric), from the "Algorithm details" of the HPA documentation.
By default the autoscaler skips scaling when the ratio of current to target metric is within 0.1 of 1.0, according to the same page. Here the queue ratio is 2.0 and the GPU ratio is 1.5, so both exceed that tolerance. Queue length proposes 8 replicas and GPU use proposes 6.1
The highest proposal is 8. A cap of 6 replicas limits the final size to 6.1 With 6 replicas the queue per pod is still about 27, above the target of 20, so requests wait longer until the load drops or the cap rises.1 Now the GPU side: if each node advertises four GPUs and a job has eight workers that each ask for one GPU, the job needs two nodes.1
| Quantity | Value (computed or illustrative) |
|---|---|
| Starting replicas | 4 |
| Queue-length proposal | 8 |
| GPU-use proposal | 6 |
| Highest proposal | 8 |
| Replica cap | 6 |
| Final replica count | 6 |
| GPUs advertised per node | 4 |
| One-GPU workers in the job | 8 |
| Nodes needed | 2 |
Text description of the diagram
- The queue-length metric proposes 8 replicas and the GPU-use metric proposes 6.
- The highest proposal is 8, and a cap of 6 replicas limits the final size to 6.
Trade-offs and edge cases
Level-based logic costs more per pass than reacting to one event, because the loop compares whole states. It buys safety. If two events arrive for one object, the second pass sees the work is done. The client-go workqueue is built for this: it does not process one item by two workers at the same time, and an item added several times before it is processed is handled once.
Text description of the diagram
- An edge-triggered handler reacts to each change it is told about.
- A level-based reconcile compares the whole current state with the desired state.
GPU sharing is a trade. A whole GPU per pod is simple and isolated. Multi-instance GPU (MIG) slices pack more small models onto one card, at the cost of fixed slice sizes (KubeNatives, secondary source). Dynamic Resource Allocation (DRA), generally available since Kubernetes v1.34 according to the DRA documentation, lets a pod claim a device by its attributes instead of by a bare count.
Metric delay is a real edge case. Google's documentation says response times over approximately 50 milliseconds would be more than typically observed with the standard metrics, which affects the recalculation period. A slow custom metric means slow scaling.
Last, percentage targets use averages. The same page says the autoscaler calculates the average utilization as a percentage of the pod's requests, so one hot pod can hide behind quiet ones. Quality problems after scaling are a different topic; see model drift and large language model (LLM) observability.
What happens when the autoscaler adds pods and no GPU is free?
The new pods stay Pending, because the scheduler finds no node with a free GPU. The HPA changes only the number of pods. Adding GPU nodes is the job of a node autoscaler such as the cluster autoscaler described in the node autoscaling documentation, which adds a node when pods cannot be placed. So the serving loop depends on two loops, and node start-up time delays the new replicas. Spare capacity kept ready trades cost for speed, and pending pods are a useful signal to watch beside queue length.
Hardware questions: GPUs per node and memory per GPU
Interview hardware questions ask how you size a node and what the scheduler can see. A node advertises a count of GPUs, such as four, and the scheduler matches pods to that count. The NVIDIA plugin reports no memory information, only a count (KubeNatives, secondary source), so the scheduler cannot tell a small card from a large one on its own.
Teams label nodes by GPU model and select them with node selectors, or use device attributes through DRA. Pick GPUs per node from the largest job that needs fast links between GPUs. Pick memory per GPU from the model's weights plus its working memory, because a pod that needs more memory than its card holds fails even when the count fits.
How to answer it in an interview
How would you use Kubernetes to run ML workloads?
Kubernetes runs on control loops. I declare the desired state, such as a training job or a model server, and controllers keep reconciling the cluster toward it. They act on current state, not single events, so a repeated or missed event is harmless. GPUs are not built in: a device plugin advertises them as an extended resource and the scheduler counts them. An operator wraps a job type in a custom resource. For scaling I use the autoscaler on queue length or latency, with a replica cap, and a node autoscaler adds GPU nodes, since new pods stay Pending without a free GPU.
The spoken structure: say the control loop, then GPUs, then operators, then autoscaling, and end with the failure you watch for.
Common mistakes
- "Kubernetes understands GPUs." (wrong) "A device plugin advertises GPUs as an extended resource, and the scheduler counts them." (right)
- "Edge-triggered code that reacts to each event once is enough." (wrong) "A controller is level-based: it reconciles current state toward desired state and can run again safely." (right)
- "Operators are a separate system." (wrong) "An operator is an ordinary controller that watches a custom resource." (right)
- "The autoscaler only scales on CPU." (wrong) "It can use custom and external metrics through an adapter, and takes the highest proposal." (right)
- "The autoscaler adds GPU capacity." (wrong) "The autoscaler adds pods; a node autoscaler adds GPU nodes, and pods stay Pending until then." (right)
Follow-up questions
What happens when a resource is changed and how do things propagate to the reconciler?
The change is written to the API, and controllers watching that kind of object are told. They then compare the object's spec with the world and act. Because the loop is level-based, a missed notification is repaired on the next pass.
How do Kubernetes operators watch and reconcile, and what should you keep in mind?
An operator watches its custom resource and the objects it creates, then drives them toward the spec. Keep each pass safe to repeat, and write the spec so the user states what they want, not steps.
How would you debug a pod that is not reachable?
Work from the inside out, in the order of the Kubernetes guide to debugging Services. Run kubectl describe pod and read the events, then check that the pod is running and its readiness probe passes, because an unready pod is left out of the Service's endpoints. Check that the Service selector matches the pod's labels and that the Service has endpoints, test that the Domain Name System (DNS) name resolves from another pod, then look at network policy and the route in.
For a custom CRD, how is a new version rolled out and rolled back?
A CRD lists its versions, each with a served flag that decides whether the API answers requests in that version, and exactly one version carries the storage flag, which is the version written to the store (CRD versioning). To roll out, add the new version as served with a conversion webhook, a service the API server calls to convert objects between versions, make the controller handle both, then switch storage to the new version and rewrite the stored objects; status.storedVersions lists every version ever stored, and an old version leaves the CRD only after every object is migrated.
To roll back, keep the old version served and in the CRD, switch storage back and revert the controller, because objects already stored in the new version still need the webhook to be read, and dropping a version that still has stored objects makes them unreadable.
If several threads reconcile events of the same object, how is the race handled?
Ensure one object is handled by one worker at a time. The client-go workqueue keys work by object, never gives one item to two workers concurrently and merges repeated adds into one, and each pass is still written to be safe to repeat.
How would you orchestrate Kubernetes clusters to run large language models efficiently?
Request whole GPUs by name, or slices where models are small, and use gang scheduling so all workers of a job start together. Autoscale serving with a controller that reads queue length, p95 and p99 latency and GPU use, takes the highest proposal, respects a replica cap and relies on a node autoscaler for new GPU nodes. A large language model (LLM) also needs traces and cost signals; see LLM observability.
- A controller moves current state toward desired state and is safe to run again.
- An operator is a controller for a custom resource and adds ML job types.
- GPUs reach the scheduler as an extended resource through a device plugin.
- The autoscaler takes the highest proposal from its metrics, limited by a replica cap.
- Late metrics slow scaling, so watch the recalculation period.
Sources
- Controllers - Kubernetes documentation — Kubernetes, 1 Sep 2024
- Operator pattern - Kubernetes documentation (v1.34) — Kubernetes, 27 Aug 2025
- Horizontal Pod Autoscaling - Kubernetes documentation — Kubernetes, 4 Aug 2026
- CNCF TAG App Delivery Operator White Paper v1.0 — CNCF, 1 Jan 2021
- Horizontal Pod autoscaling concepts - Google Kubernetes Engine docs — Google Cloud, 9 Oct 2026
- Device plugins - Kubernetes documentation — Kubernetes, 26 Oct 2026
- Schedule GPUs - Kubernetes documentation — Kubernetes, 26 Oct 2026
- Dynamic Resource Allocation - Kubernetes documentation — Kubernetes, 26 Oct 2026
- Versions in CustomResourceDefinitions - Kubernetes documentation — Kubernetes, 26 Oct 2026
- Debug Services - Kubernetes documentation — Kubernetes, 26 Oct 2026
- Node Autoscaling - Kubernetes documentation — Kubernetes, 26 Oct 2026
- workqueue package - client-go — Kubernetes (client-go), 26 Oct 2026
- How Kubernetes Schedules GPUs (secondary) — KubeNatives, 7 Mar 2026
- GPUs in Kubernetes: how it actually works (secondary) — ScaleOps, 21 Jul 2026
