How do you design on-device and privacy-constrained ML?

In this article

1. Before you start
2. Why interviewers ask this
3. On-device and privacy-constrained ML from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the ML system design interview path.

Why interviewers ask this

A privacy limit changes every part of an ML system: where data lives, how the model learns, how big it can be and what happens when the device is too weak. Candidates report prompts of this kind in design rounds at large consumer-device companies: an on-device recommender for a news app, a federated speech system with stated privacy parameters, a router between device and cloud, and a feed ranker under financial rules.

A weak answer says "train on the device" and stops. A strong answer states the threat, names the privacy unit, explains how learning still happens without raw data and prices the accuracy cost. A Staff-level answer also knows what the privacy method does not protect, such as a server that must still be trusted, and designs the fallback for requests the device cannot serve.

On-device and privacy-constrained ML from first principles

Differential privacy

Differential privacy is a mathematical definition of privacy under which a system's output barely changes when any one unit of data is added or removed, usually achieved by adding carefully calibrated noise that averages out across many people.

Privacy unit

The privacy unit is the thing a privacy method protects: a single training example, or everything one user contributes. It must be chosen before the noise is set.

In plain words

Think of a teacher who wants to learn which exam questions confuse a class. She does not want to read anyone's notebook. Each student works out a short note, such as "question 4 was hard". The teacher averages the notes. No notebook leaves the desk. The analogy breaks in one place. A note can still reveal something about its author. So the teacher asks students to blur their notes a little. That blur costs accuracy.

In short, the system keeps data and the personalized model local. It learns shared patterns from clipped and noised updates. It uses a server only for work the device cannot do.

Step 1: Keep raw data and the personal model on the device

On-device ML keeps raw data and the personalized model on the user's phone, which removes the central store an attacker would target. Apple Security Research (June 2024) puts it this way: data that exists only on devices is disaggregated and has no centralized point of attack.

McMahan et al. list when the approach fits: on-device data is better than proxy data in a data center, the data is sensitive or large, and labels come naturally from user interaction. A news-reading app fits all three, because a tap or a read is the label.

Input: user events such as reads and taps. Output: a local feature store and a local ranking model. Raw data stays on the device. Note that "on-device" does not mean "no server": models still need training, distribution and updates, which the next steps cover.

Step 2: Learn together with federated averaging

Federated learning lets devices improve a shared model without sending raw data. The server sends the current model to a sample of devices, each device trains on its own data, and the server averages the returned updates. McMahan et al. call this the FederatedAveraging algorithm, called FedAvg: local stochastic gradient descent on each client plus model averaging on the server.

Communication is the main cost. The paper reports that FedAvg cut the required communication rounds by 10 to 100 times against synchronized stochastic gradient descent, the FedSGD baseline with one gradient step per round. Devices send model updates, not raw data.

Input: the current global model. Output: an averaged model. One limit: the paper states that some trust in the coordinating server is still required, so federated learning narrows the attack surface to the device but does not make the system private by itself.

Step 3: Differential privacy protects a chosen unit: an example, an event or a user

Differential privacy limits what a model's output reveals about any one privacy unit, so choose the unit deliberately; user-level is the stronger claim. Google Research (2022) explains that example-level protection is not necessarily strong enough when users contribute many examples, because one user's data can still be memorized. User-level protection requires that the output distribution does not change if all of one user's examples are added or removed, and Google's differentially private federated learning work targets it. The recipe is clipping each user's contribution and then adding noise. The same post reports a production next-word model for Spanish-language Gboard users trained with a formal bound of ρ=0.81 in zero-concentrated differential privacy (zCDP).

Also choose local or central. Apple describes central differential privacy as needing raw data on the server, and chose the local setting so raw data is never collected. Apple's local system protects events, not users: its page describes event-level differential privacy, where an event might be a user typing an emoji. That weakens the per-user claim when one user reports many events.

What ε, δ and ρ mean

Privacy claims quote three symbols. ε (epsilon) bounds how much any one unit can shift the probability of an output, by a factor of at most $e^{\epsilon }$; smaller is stronger. δ (delta) is a small probability that this bound fails. The zCDP parameter ρ (rho) is a different accounting scale, so ρ=0.81 is not an ε. A reported Siri prompt states $\epsilon =7.2$ and $\delta ={10}^{-9}$; those numbers come from the question, not from a benchmark.

How differentially private federated averaging clips, adds noise and counts the budget

The Siri prompt asks about clipping and noise, so here is the mechanism, following McMahan et al. (2018). Each sampled device's update is clipped to an L2 norm bound S, the usual vector length, so no user can move the model by more than S. The server adds Gaussian noise with standard deviation $\sigma S$ to the sum of the sampled updates. Sampling only some devices each round amplifies privacy.

An accountant, such as a Rényi or zCDP accountant, converts σ, the sampling rate and the round count into a final ε and δ. Per-layer clipping, the variant the prompt names, gives each layer its own bound so one large layer does not dominate. Clipping and noise both cost accuracy. Secure aggregation is a cryptographic protocol that lets the server see only the sum of many devices' updates, never one update (Bonawitz et al., 2017); it answers "the server must still be trusted" for individual updates.

Step 4: Shrink the model so it fits the device

A device has a small memory, a tight battery and a latency target, so the model must be compressed. Quantization is storing each model number in fewer bits, for example 8-bit integers instead of 32-bit floats, which cuts the size to a quarter at some cost in accuracy. A smaller architecture, trained to imitate a larger one, is the other common tool.

State the budget before choosing tools. A reported prompt asks for under 200 milliseconds per recommendation for 50 million users; that is a requirement of the question, so split it into stages as the worked example does. This page leaves out vendor-specific memory and battery figures, because none was verified for it: say in the interview that you would measure them on the target hardware.

Input: a trained model and a hardware budget. Output: a smaller model that meets the latency target. Measure accuracy before and after compression, because a model that is fast but wrong is not a win.

Step 5: Send only hard requests to a private cloud tier

Some requests need a model too big for any phone, so the design needs a cloud tier with its own privacy rules. Apple Security Research notes that large cloud models need unencrypted access to the user's request, so encryption alone cannot protect it. Its Private Cloud Compute design therefore uses the request only to answer it and does not retain it, not even in logs or for debugging.

The post lists five requirements: stateless computation, enforceable protections, no privileged runtime access, non-targetability and verifiable transparency. Non-targetability means an attacker cannot go after specific users without compromising the whole system.

The post does not describe how requests are routed between device and cloud, so the routing logic is yours to design: a sensitivity label that keeps personal requests local, and a difficulty estimate that sends only what the local model cannot answer.

1 · Keep data local2 · Federated learning3 · User-level noise4 · Compress5 · Private cloudReplay

Step-by-step description

1. Raw data and the personal model stay on the device.
2. Devices send updates, and the server averages them.
3. Clip each user's update, then add noise.
4. A smaller model meets the latency budget.
5. Only hard requests leave the device.

• Raw data stays on the device.
• Devices send model updates, not raw data.
• The recipe is clipping each user's contribution and then adding noise.
• Send only hard requests to a private cloud tier.

Worked example

Take an on-device recommender with the scale of a reported prompt: 50,000,000 people. Every other number below is an illustrative assumption, computed in a script.¹

A model of 5,000,000 numbers takes 20 megabytes at 32 bits and 5 megabytes at 8 bits. If a round samples 10,000 devices, one round uploads 200 gigabytes of updates at 32 bits and 50 gigabytes at 8 bits, and a given device is sampled about once every 5,000 rounds on average.¹ Federated averaging helps the round count: in Table 2 of McMahan et al. a handwritten-digit convolutional network needed 626 rounds with FedSGD to reach 99 percent accuracy and 29 with FedAvg (5 local epochs and batch size 50) on an independent and identically distributed (IID) split, a 21.6 times reduction.¹

On a non-IID split, where each device holds a skewed slice, the counts for that setting were 483 and 334, only 1.4 times.¹ Across the settings in Table 2, the non-IID gain ranged from slower than FedSGD to 2.8 times (173 rounds), against up to 34.8 times on IID data.¹ Non-IID data is data whose distribution differs from device to device, and it is the normal case on phones.

In that one setting, FedAvg needed 29 rounds against 626 for FedSGD on an IID split, and 334 against 483 on a non-IID split.¹

Text description of the diagram

• FedAvg needed 29 rounds against 626 for FedSGD on an IID split.
• FedAvg needed 334 rounds against 483 for FedSGD on a non-IID split.

For the 200-millisecond target, an illustrative split is 30 milliseconds for feature reads, 50 for candidate retrieval, 70 for the ranking model, 20 for post-processing and 30 spare.¹

Noise averaging can be shown with randomized response: each user reports the truth with probability 0.75, which gives $\epsilon =\ln \frac{p}{1-p}=\ln 3\approx 1.10$ for $p=0.75$, and otherwise lies. With 1,000 respondents the estimate of a true 30 percent rate is off by about 3.1 percentage points; with 1,000,000 respondents, about 0.098.¹ This is why local noise is usable only at scale.

Trade-offs and edge cases

Every privacy tool trades something. The table shows where trust and cost move.

Text description of the diagram

• Central differential privacy needs a collector that holds raw data and must be trusted.
• Federated learning keeps raw data local but still needs trust in the coordinating server.
• Cloud inference needs the request in plaintext.

Central differential privacy needs a collector that holds raw data and must be trusted. Local differential privacy needs no trusted collector, but Apple's analysis frames a trade-off among utility, privacy, server computation and device bandwidth. Federated learning keeps raw data local but, as McMahan et al. state, still needs trust in the coordinating server; formal protection adds clipping and noise, with an accuracy cost. Cloud inference gives the most capable models but needs the request in plaintext.

Edge cases: a cold-start user has no local history, so the first recommendations use a global model; a weak or old device may be unable to run the model at all, so the router must degrade gracefully; and heavy users dominate unless each user's contribution is clipped. Reported practice differs on which techniques to combine, so state your assumptions. Secure aggregation narrows what the server sees of each update, but it does not by itself bound what the summed result reveals; that is the job of the noise.

How to answer it in an interview

The spoken structure: say where data lives, name the privacy unit, explain how learning works, give the cost, then the fallback.

Common mistakes

• "Federated learning is private by itself." (wrong) The server must still be trusted and updates can leak. (right) "Federated learning keeps raw data local; I add clipping and noise for a formal claim."
• "On-device means no server." (wrong) Models still need training, distribution and updates. (right) "The server coordinates and ships models; it never needs raw data."
• "Differential privacy is free." (wrong) Clipping and noise cost accuracy. (right) "I would measure the privacy-utility trade-off and report it."
• "Example-level noise protects the user." (wrong) A user with many examples can still be memorized. (right) "I choose the privacy unit deliberately; user-level is the stronger claim."
• "Non-IID data is a minor detail." (wrong) It is the usual case and shrinks federated averaging's gains. (right) "I expect fewer savings and test on skewed splits."

Follow-up questions

How does federated learning work?

The server sends a model to a sample of devices, each trains locally and returns an update, and the server averages the updates into a new model. Raw data never leaves the device; only updates do.

What is the difference between local and central differential privacy?

In central differential privacy a server collects raw data and adds noise to results; in local differential privacy each device adds noise before sending anything. Apple chose the local setting so raw data is never collected.

Is federated learning private on its own?

No. The coordinating server must still be trusted, and updates can leak information, so a formal claim needs differential privacy on top; secure aggregation hides individual updates from the server.

How do you decide what runs on the device and what goes to the cloud?

Keep sensitive, latency-critical and simple requests on the device, and send only requests the local model cannot answer to a cloud tier that does not retain them. The routing rule is a design choice you state and test.

How would you design a feed ranking system under financial regulatory constraints?

Ask which rules apply instead of assuming them, then turn each into a requirement. In general terms, such requirements concern how long data is retained, an audit trail of what was shown and why, and being able to explain what a user sees. The ranking itself follows the search ranking design pattern, and real-time fraud detection and ML for healthcare show the same habit of treating rules as design inputs.

How would you design a small language model that runs on a phone?

Fit the model to the memory and latency budget with quantization and a smaller architecture trained to imitate a larger one, as in Step 4, and send requests it cannot answer to a cloud tier. Next-word suggestion is the natural use, as in typeahead design; measure quality and tone on the target device before shipping.

Sources

1. Communication-Efficient Learning of Deep Networks from Decentralized Data — PMLR / AISTATS (McMahan et al.), 20 Apr 2017
2. Private Cloud Compute: A new frontier for AI privacy in the cloud — Apple Security Research, 10 Jun 2024
3. Learning with Privacy at Scale — Apple Machine Learning Research, 1 Dec 2017
4. Federated Learning with Formal Differential Privacy (Google Research blog) — Google Research (McMahan and Thakurta), 28 Feb 2022
5. Learning Differentially Private Recurrent Language Models (arXiv 1710.06963) — arXiv / ICLR 2018 (McMahan et al.), 18 Oct 2017
6. Practical Secure Aggregation for Privacy-Preserving Machine Learning — Google Research (Bonawitz et al., CCS 2017), 30 Oct 2017