What is edge AI MLOps, and how do you run it?

In this article

1. Before you start
2. Why interviewers ask this
3. Edge AI MLOps from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the MLOps learning path.

Why interviewers ask this

A model on a device keeps running long after the release, on hardware the team does not control. Edge AI MLOps questions test whether a candidate can operate a model after the first release, not only build it.

Cloud MLOps assumes one environment that you can redeploy quickly. A device fleet breaks that assumption: the hardware and operating systems differ, and rolling back to an earlier version after a production problem adds another layer of complexity (EdgeMLOps, arXiv 2501.17062).

A weak answer says: convert the model and copy it to the device. A strong answer shows the whole loop. It compresses the model and re-checks accuracy, packages a version, delivers it in a controlled way, releases it in waves with a rollback path, and monitors the fleet to decide when to retrain. A Staff-level answer also names what is vendor-specific and what is general, and says which numbers it would measure rather than assume.

Edge AI MLOps from first principles

Edge AI MLOps

Edge AI MLOps is the set of practices that take a machine learning model trained in the cloud and deliver, run, monitor, update and roll it back on a fleet of devices, where the hardware differs from device to device and connections come and go.

Staged rollout

A staged rollout is a release method that sends a new model version to a small group of devices first, checks that they stay healthy, and then widens the release in waves, with a rule that stops the rollout if failures pass a set limit.

In plain words

Picture a restaurant chain. Head office writes the recipe and tests it in one kitchen. Each branch cooks it on its own oven, and some branches lose their phone line for hours. Head office must send the recipe in a form each oven handles, update branches without breaking dinner, and hear back when a dish goes wrong. The analogy breaks in one place: a branch can close for an afternoon, but a device usually keeps serving while it updates.

In one precise sentence, edge AI MLOps is cloud MLOps whose deployment target is a heterogeneous, intermittently connected fleet instead of one cloud service (Google Cloud, last reviewed August 2024; EdgeMLOps, arXiv 2501.17062).

MLOps is, in Google's words, "an ML engineering culture and practice that aims at unifying ML system development (Dev) and ML system operation (Ops)". Its loops are continuous integration (CI), continuous delivery (CD) and continuous training (CT). Continuous training is automatically retraining candidate models for testing and serving, and Google Cloud calls it a property unique to ML systems. "Edge" does not mean "no cloud": training and heavy processing stay in the cloud, and inference runs locally for low latency and cost savings (Greengrass developer guide).

Step 1: Compress the model and re-check its accuracy

Compression shrinks a trained model so that it fits and runs on a small device, and its accuracy must be measured again afterwards. Input: the full-precision model from the cloud. Output: a smaller model and an accuracy report on the target.

Quantization is storing weights, and sometimes the numbers passed between layers, with fewer bits. Google AI Edge says post-training quantization can reduce size and improve latency with little accuracy loss. Float16 is about two times smaller. Dynamic range quantization is about four times smaller with a two to three times speedup on a central processing unit (CPU). Full integer quantization is also about four times smaller, speeds the CPU up three times or more, and is the form that the Edge tensor processing unit (TPU) and microcontrollers need. It needs a representative sample of around 100 to 500 examples.

Quantization-aware training (QAT) retrains for the smallest accuracy loss but needs labelled data (model optimization). Google's guidance says it is important to check the accuracy of the quantized model. A compressed model must have its accuracy re-measured before it ships.

Step 2: Package the model as a versioned component

Packaging wraps the compressed model and its runtime into a versioned unit that a device can install. Each model build ships as a versioned package for its hardware type. Input: the checked model. Output: an artifact with an identifier that the cloud and the device both recognize.

On Greengrass you deploy the model as a machine learning component, and deploying the inference component also deploys its model and runtime dependencies (Greengrass developer guide). Hardware and operating system differences are a named challenge of edge fleets (EdgeMLOps), so a fleet often needs one build per hardware type. A version identifier lets you answer "which model is running on which device?", which every later step depends on. The same guide lists a minimum of 500 MB of local storage for the Amazon Web Services (AWS) sample components, so check the device budget before choosing a model.

Step 3: Deliver the update by pull or push

Delivery is the over-the-air (OTA) step that moves the new package from the cloud to each device. Devices either pull a new version or the server pushes it. Input: a package and a list of target devices. Output: devices holding the new version.

Pull-based updates mean the device fetches new model versions from a central server. Push-based updates mean the server pushes them to the device. The same source says updates should happen within predictable, controlled time windows. Because devices are intermittently connected, a pushed update may not arrive when sent, and a pulled one arrives only when the device checks in. Total bandwidth is model size times device count, so Step 1 saves money here as well as storage; the worked example below computes it.

Step 4: Stage the rollout and roll back on failure

A staged rollout limits the damage of a bad model by releasing it in waves. A staged rollout widens in waves and rolls a failed device back. Input: a package and a wave plan. Output: a fleet that moved forward, or stopped and returned to the previous version.

Staged rollout is a platform feature, and Greengrass is one example. Its job configuration sets rollout, timeout and abort settings; the rollout configuration "defines the rate at which the job rolls out to the fleet" with an exponentialRate and a baseRatePerMinute, and abortConfig defines when and how to stop. On the device, the failure-handling policy set to ROLLBACK (roll back to the last configuration) returns the device to its previous configuration when a deployment fails, while the DO_NOTHING setting keeps the changes; a failed bootstrap deployment rolls back even then (deployment failure handling). Take the default and maximum rates from the documentation of your platform.

Step 5: Monitor the fleet and retrain

Monitoring tells you whether the deployed model still works, and the answer feeds retraining. Fleet monitoring decides when to retrain. Input: health and quality signals from devices. Output: a decision to hold, roll back, or retrain in the cloud.

The EdgeMLOps paper names OTA updates and centralized monitoring as essential for keeping model performance up in changing edge environments (arXiv 2501.17062). A model goes stale when live data drifts, as in trained in January, bad in July; that is where continuous training closes the loop. No primary source in this page's research sets what edge telemetry to send. A sound design sends small summaries, such as error counts and score distributions, rather than raw user data. For the cloud-side signals, see large language model (LLM) observability, and for turning feedback into new training data, build a data flywheel from production feedback.

1 · Compress2 · Package3 · Deliver4 · Roll out5 · MonitorReplay

Step-by-step description

1. Shrink the model and re-check accuracy.
2. Ship a versioned package per hardware type.
3. Devices pull, or the server pushes.
4. Widen in waves; roll back on failure.
5. Fleet signals decide when to retrain.

• A compressed model must have its accuracy re-measured before it ships.
• Each model build ships as a versioned package for its hardware type.
• Devices either pull a new version or the server pushes it.
• A staged rollout widens in waves and rolls a failed device back.
• Fleet monitoring decides when to retrain.

Worked example

Take an illustrative fleet of 10,000 devices and a 40 MB full-precision (fp32) model. Float16 is about two times smaller and int8 about four times smaller, so the model becomes 20 MB and 10 MB.¹ A full push of the model to the fleet moves 400 GB at fp32, 200 GB at float16 and 100 GB at int8.¹ Compression cuts the bytes sent to every device.

If the new model is bad, a three-wave plan exposes 100 devices before the first health check, instead of 10,000.¹ The int8 push of the first wave costs 1 GB.¹ A delta update, which sends only the changed parts, would cut the totals further, but this page has no primary figure for that saving, so the table does not assume one. Sizes in GB use 1 GB as 1,000 MB.

Text description of the diagram

• A full push of the model to the fleet moves 400 GB at fp32, 200 GB at float16 and 100 GB at int8.
• Compression cuts the bytes sent to every device.

Trade-offs and edge cases

Compression trades accuracy and effort against size and speed. Float16 and dynamic range quantization need no calibration data, and integer quantization needs a small unlabelled sample. Float16 is about two times smaller with graphics processing unit (GPU) acceleration, and full integer quantization is about four times smaller and runs on the Edge TPU and microcontrollers.

Text description of the diagram

• Float16 is about two times smaller with graphics processing unit acceleration.
• Full integer quantization is about four times smaller and runs on the Edge TPU and microcontrollers.

Other cases to name in a design:

• Cloud versus device. Local inference gives low latency and cost savings, but training and heavy processing stay in the cloud (Greengrass guide). For when a result must be immediate or can wait, see batch, streaming and real-time inference.
• Rollback is harder than in the cloud. A fleet is mixed and partly unreachable, so some devices will not have the bad version yet and others will not get the rollback yet (EdgeMLOps). Version identifiers tell you which is which.
• Vendor ranges are not promises. The speedups quoted by Google AI Edge are ranges, so measure on your own hardware.
• Robots and embedded devices add power, storage and safety limits; see deploying models to robots and embedded devices.

How to answer it in an interview

The spoken structure: say where training and inference run, then compress and re-check, then version, deliver, roll out in waves with rollback, and close with monitoring and retraining.

Common mistakes

• "Quantization is free." (wrong) It can lose accuracy, and the loss must be re-measured. (right) "I compress, then re-check accuracy on the target; QAT is the option when the loss is too high."
• "Edge means no cloud." (wrong) Training and heavy processing stay in the cloud. (right) "Training is in the cloud and inference is on the device."
• "Dynamic range quantization is fully integer." (wrong) Activations are still stored in floating point. (right) "For integer-only hardware I use full integer quantization with a calibration sample."
• "Rolling back a model is like rolling back a cloud service." (wrong) A fleet is mixed and partly unreachable. (right) "I track versions per device and set a failure policy that restores the previous one."
• "Deploy means done." (wrong) Models go stale as data drifts. (right) "I monitor the fleet and retrain through continuous training."

Follow-up questions

How do you roll back a bad model on edge devices?

Keep the previous version installed, and set a failure policy so a failed update returns the device to it, such as the rollback setting on Greengrass. For a bad model that installs but performs badly, an abort condition stops the rollout, and a new deployment of the old version restores devices.

Should devices pull a new model or should the server push it?

Both exist. Pull-based means the device fetches the new version, and push-based means the server sends it (Microsoft data science blog). Whichever you pick, use controlled time windows, and plan for devices that were unreachable when the update went out.

Do you still need the cloud if inference runs on the device?

Yes. The cloud trains the model, stores artifacts and does heavy processing, while the device runs inference locally (Greengrass guide).

How do you know when to retrain an edge model?

Watch for drift in the summaries devices send back, and for a fall in a quality metric you can compute. When it crosses a threshold you set, start continuous training and send the new model through the same compress, check and staged-rollout steps.

When is quantization-aware training worth it?

When post-training quantization loses too much accuracy and you have labelled training data. QAT retrains for the smallest accuracy loss (model optimization), at the price of a training run.

Sources

1. MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center, 28 Aug 2024
2. Key requirements for an MLOps foundation — Google Cloud Blog, 26 Oct 2026
3. Post-training quantization — Google AI Edge, 26 Oct 2026
4. Post-training quantization (representative dataset detail) — Google AI Edge, 26 Oct 2026
5. Model optimization — Google AI Edge, 26 Oct 2026
6. Post-training dynamic range quantization — Google AI Edge, 26 Oct 2026
7. Perform machine learning inference — Amazon Web Services documentation, 26 Oct 2026
8. Local deployments: component deployment and failure handling — Amazon Web Services documentation, 26 Oct 2026
9. create-deployment (Greengrass V2 CLI reference) — Amazon Web Services documentation, 26 Oct 2026
10. EdgeMLOps: Operationalizing ML models with Cumulocity IoT and thin-edge.io for Visual Quality Inspection (arXiv 2501.17062) — arXiv, 1 Jan 2025
11. Edge deployment monitoring: from cloud prototyping to edge execution — Data Science at Microsoft, 26 Oct 2026