Deploying Models to Robots and Embedded Devices

In this article

1. Before you start
2. Why interviewers ask this
3. Deploying models to robots and embedded devices from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the machine learning operations (MLOps) interview questions path and assumes the basics in machine learning algorithms.

Why interviewers ask this

A model that is accurate on a cloud server can be useless on a robot if it misses its deadline or does not fit in memory. Candidates report three versions of this question: optimize a trained model for an edge device with limited compute, optimize a large PyTorch model for edge hardware, and walk through optimizing a U-Net for an edge device, including why you would choose one quantization method over another.

A weak answer says "quantize it" and stops. A strong answer gives an ordered pipeline, names what each step changes, and says how you would check that accuracy and speed survived. A Staff-level answer adds the constraint that is specific to robots: the model must meet a deadline every time, so you measure on the target device and care about the slowest run, not the average.

Deploying models to robots and embedded devices from first principles

Edge device

An edge device is a computer that runs a model next to where the data is produced, such as a robot, a camera or a microcontroller board. It has fixed memory, compute and power budgets and often no dependable connection to a data center, so the model must fit and run locally.

Operator fusion

Operator fusion is a graph optimization that merges several neighboring operations, such as a convolution followed by an addition, into one operation. The runtime then makes fewer passes over memory and has fewer steps to schedule.

Quantization-aware training

Quantization-aware training (QAT) is a training method that simulates low-precision rounding in the forward pass, so the weights learn to tolerate the rounding error that quantization causes.

In plain words

Think of packing a suitcase for a flight with a strict weight limit. You remove what you do not need. You fold the rest tightly and swap heavy items for light ones. Then you weigh the suitcase at the airport, not at home. The analogy breaks in one place. A scale shows weight at once. Nothing shows how much accuracy you lost until you test the packed model on real data.

In precise terms, deployment to an edge device is a pipeline. It converts a model to a portable format and rewrites its graph to do less work. It then lowers the precision of its numbers. Last, it verifies accuracy and speed on the target hardware.

Step 1: Export the model to a portable format

The ONNX format (see the ONNX project) is a framework-neutral file that stores a model's graph of operations and its weights, so one runtime can run a model trained in PyTorch or TensorFlow. The ONNX Runtime documentation describes models that are exported from PyTorch or converted from TensorFlow.

Input: the trained model. Output: one file with the graph and weights. After exporting, run the same test inputs through both versions and compare outputs; a mismatch here is a bug in the export, not in quantization. For a U-Net, also check that every operator is supported by the runtime on your target device. You now have a file any compatible runtime can load, which the next step rewrites.

Step 2: Optimize the graph

Graph optimization rewrites the model's graph so it computes the same result with less work. The ONNX Runtime graph-optimization page lists constant folding, which computes parts of the graph that depend only on constants ahead of time, and fusions such as folding an add into the bias of a convolution.

Its basic rewrites run before the graph is split across execution providers, so they apply everywhere. Its extended fusions, such as matrix-multiply plus add, run after that split and only on nodes assigned to certain providers. Input: the exported graph. Output: a smaller, fused graph with the same outputs. Because these rewrites are meant to preserve results, you still re-run your test inputs to confirm it.

Step 3: Quantize, then check accuracy and speed

Quantizing stores weights as lower-precision numbers, most often 8-bit integers instead of 32-bit floating point. ONNX Runtime supports that 32-bit to 8-bit conversion through Python interfaces, and the LiteRT pages also list 16-bit float quantization. The easy route is called post-training quantization (PTQ): you convert an already-trained model with no retraining (Google AI Edge).

PTQ comes in flavors. Dynamic range quantization needs no calibration data. Calibration data is a small set of representative inputs run through the model to learn the range of the values inside it. Full integer quantization uses that sample and is needed for integer-only hardware such as the Edge tensor processing unit (TPU) and microcontrollers, per the same Google page. If neither PTQ method meets your accuracy goal, ONNX Runtime suggests QAT, which means retraining in the original framework.

The check matters. The LiteRT benchmark table comes from TensorFlow Lite and was measured on Pixel 2 devices using a single big core central processing unit (CPU), not with ONNX Runtime. In it, MobileNet-v2 top-1 accuracy is 0.719 originally, 0.637 after PTQ and 0.709 after QAT.¹

Step 4: Measure on the target device

Measurement on the real device decides whether the earlier steps worked, because a smaller model is not always a faster one. In the same table, measured on that phone CPU, MobileNet-v2 latency is 89 milliseconds originally and 98 milliseconds after PTQ, which is slower, and 54 milliseconds after QAT.¹ The source does not explain the cause, so do not guess it; the table shows only that speed depends on the hardware and the supported kernels.

For a robot, measure the slowest runs, not the average, because a control loop that misses its deadline once is a failure. Record accuracy on a held-out set, memory used, latency across many runs and power or temperature on the device itself. Treat any published number as one reported case, not a prediction for your board.

Robots add limits that a short benchmark hides. A long run can heat the board until it lowers its clock speed, so test over a sustained period. Design a safe fallback for a missed deadline, such as holding the last command or switching to a smaller backup model.

1 · Export2 · Optimize graph3 · Quantize4 · MeasureReplay

Step-by-step description

1. Export the trained model to ONNX.
2. Fuse operators and fold constants.
3. Store weights as integers, then check accuracy.
4. Test speed and accuracy on the device.

• Export the model to ONNX first.
• Optimize the graph, then quantize the weights.
• Measure on the target device last.

When quantization is not enough: prune, distill, or change the model

Pruning is a compression technique that removes weights, channels or layers that add little to a model's output. Unstructured pruning zeroes individual weights, which mostly shrinks the stored file because dense hardware still multiplies the zeros. Structured pruning removes whole channels or filters, which makes the layers themselves smaller and so can speed up dense hardware. Knowledge distillation is a training method that teaches a small student model to imitate a large teacher model, so you deploy the student.

A smaller architecture designed for edge devices, such as MobileNet, starts cheap instead of shrinking something large. The LiteRT model-optimization page lists pruning and clustering beside quantization as separate techniques. You can also compile for the target accelerator: ONNX Runtime execution providers let it run supported parts of a graph on a vendor software development kit (SDK), a graphics processing unit (GPU) or a neural processing unit (NPU) instead of the CPU. These levers stack, so re-measure on the device after each one.

Worked example

Start with size. A weight stored as a 32-bit float takes 4 bytes and as an 8-bit integer takes 1 byte. For an illustrative U-Net with 30,000,000 weights, that is 120 megabytes in 32-bit form, 60 megabytes in 16-bit form and 30 megabytes in 8-bit form.¹ The parameter count is illustrative, not the count of any specific U-Net.

Next, the mapping. ONNX Runtime's quantization formulas rest on a scale and a zero point. Take illustrative values in the range -1.0 to 3.0 mapped onto the integers 0 to 255. The scale is 4 / 255, about 0.0157, and the zero point is 64. The value 0.5 becomes the integer 96, and mapping 96 back gives 0.502, an error of about 0.002.¹ That small rounding error, repeated across millions of weights, is what quantization costs.

The two models behave differently. Inception_v3 loses little under PTQ while MobileNet-v2 loses much more, so no single method is "enough" for every model.¹ The last row shows why deadlines matter: a model plus its sensor and control code must all fit inside one cycle.

Text description of the diagram

• MobileNet-v2 top-1 accuracy is 0.719 originally.
• It is 0.637 after PTQ and 0.709 after QAT.

Trade-offs and edge cases

Each quantization option trades effort for accuracy. Dynamic range quantization is the quickest to try and, per the LiteRT documentation, needs no data. Integer quantization needs an unlabeled representative sample and suits hardware such as the Edge TPU. QAT costs a retraining run but recovered most of the lost accuracy in the benchmark above. QAT retrains the model, so it needs training data.

Text description of the diagram

• Dynamic range quantization needs no calibration data.
• Full integer quantization needs a representative sample.
• QAT retrains the model, so it needs training data.

Edge cases to name in an interview:

• Model type. ONNX Runtime recommends dynamic quantization for recurrent and transformer-based models and static quantization for convolutional ones (ONNX Runtime).
• Where QAT happens. ONNX Runtime does not retrain, so you retrain in PyTorch or TensorFlow and export again.
• Not fully integer. Dynamic range quantization leaves some math in floating point, so it does not suit integer-only targets.
• Provider-specific fusions. Extended fusions apply only to some execution providers, so the same file can run differently on two devices.
• Other levers. Pruning, distillation, a smaller architecture and an accelerator-specific execution provider stack with quantization; see the section on them above.

After deployment, the model still drifts; see model drift explained, and compare serving styles in batch vs streaming vs real-time inference.

How to answer it in an interview

The spoken structure: name the ordered pipeline, say what each step changes, then say how you verify on the target device.

Common mistakes

• "Quantization always speeds up inference." (wrong) In the LiteRT table MobileNet-v2 ran slower after PTQ. (right) "I expect a smaller model, and I measure speed on the target device."
• "Post-training quantization is always enough." (wrong) One benchmark model lost about eight points. (right) "I try PTQ first and move to QAT if it misses the accuracy goal."
• "ONNX Runtime can do the QAT for me." (wrong) It does not retrain. (right) "I retrain in the original framework and export again."
• "I only report average latency." (wrong) A robot's control loop fails on the slow run. (right) "I report the slowest runs against the deadline."

Follow-up questions

How would you optimize a large PyTorch model to run on edge hardware?

Export it to a portable format, fuse the graph, quantize, and verify output and accuracy at each step. If it is still too large, add structured pruning, distill it into a smaller student or switch to an edge-designed architecture, and compile for the device's accelerator. Then re-measure on the device.

Why would you choose QAT over PTQ for a specific model?

Choose QAT when both PTQ methods miss your accuracy goal, as ONNX Runtime advises. In the benchmark above, MobileNet-v2's accuracy drop shrank from 0.082 to 0.010 with QAT.¹ The price is a retraining run.

Does quantization always make a model faster?

No. The MobileNet-v2 PTQ result ran slower than the original, so measure on the target hardware and runtime before you promise a speedup.

What changes for a U-Net?

The pipeline is the same, but segmentation outputs can be sensitive to rounding, so check task metrics, not just file size, and confirm every operator is supported on the device. Skip connections and upsampling or transposed-convolution operators are the usual ones to check, because they can be unsupported or quantize poorly. The standard mixed-precision answer is to keep sensitive layers, often the first and last, in higher precision.

How do you update and monitor a model on a fleet of robots?

Version each model and ship it over the air to a few robots first. Widen the rollout in stages, and keep the previous version on the device so you can roll back if a check fails. Log a sample of inputs and outputs on the device, and under bandwidth limits upload summaries or flagged cases instead of raw streams. Compare them for the shifts described in model drift explained, and fit the release steps into a continuous integration, delivery and training design for machine learning (ML) apps.

Sources

1. ONNX: Open Neural Network Exchange — ONNX project, home page, accessed 2026-10-26, 26 Oct 2026
2. ONNX Runtime execution providers — ONNX Runtime (Microsoft), documentation, accessed 2026-10-26, 26 Oct 2026
3. Quantize ONNX models — ONNX Runtime (Microsoft), documentation, accessed 2026-10-26, 26 Oct 2026
4. Graph Optimizations in ONNX Runtime — ONNX Runtime (Microsoft), documentation, accessed 2026-10-26, 26 Oct 2026
5. Post-training quantization — Google AI Edge (LiteRT), documentation, accessed 2026-10-26, 26 Oct 2026
6. Model optimization — Google AI Edge (LiteRT), documentation, accessed 2026-10-26, 26 Oct 2026