In this article
Before you start
This page sits under the large language model (LLM) evaluation guide and assumes you know roughly how a model is trained; the transformer overview covers that background.
Why interviewers ask this
A model that leaks one customer's phone number is a product incident, and a deletion request that the system cannot honor is a legal one. Interviewers therefore use privacy to test whether a candidate treats a trained model as a store of data, not only as a function. The topic is reported by candidates in questions about regulated customers, deletion requests under the General Data Protection Regulation (GDPR), and sensitive data found in a model's output.
A weak answer says "we anonymize the data" or "we use federated learning." A strong answer separates three places where data lives: stored copies, training sets and the weights themselves. It names a control for each and says what each one costs. A Staff-level answer adds how you would detect a leak with evaluations, because a privacy control you cannot test is only a hope.
Privacy in ML systems from first principles
- Training data extraction
- Training data extraction is an attack in which someone who can only send prompts to a language model recovers individual examples that the model saw in training, by generating many outputs and picking the ones that look memorized rather than composed.
- Federated learning
- Federated learning is a way of training one shared model in which the data stays on many devices or servers, each computes a model update locally, and only those updates are sent to a central server and combined.
In plain words
Picture a student who studies for an exam by reading a thick folder of notes. Most of what they keep is the general idea. A few lines, such as a striking phone number, stay word for word. If you ask the right question, they recite it. The analogy breaks in one place: you can ask a student to forget a page, but a trained model has no page to remove, because every training example nudged the same shared weights.
In one precise sentence, privacy in ML systems means bounding what the data, the training process and the trained model reveal about any one individual. Three controls follow. Delete and govern the data you store, add noise so no single person moves the result much, and test the model for leaks before and after release.
Step 1: Models can memorize and reveal their training data
A model can reproduce training text word for word, so a user with nothing but query access can pull that text out. Carlini et al. (2021) attacked the Generative Pre-trained Transformer 2 (generative pre-trained transformer (GPT)-2) language model this way and report that they could extract hundreds of verbatim sequences from its training data. The extracted material included personally identifiable information (PII) such as names, phone numbers and email addresses, plus code and conversations, and it was public web data.
Two findings matter for interviews. The attack worked even though each extracted sequence appeared in only one training document, so rarity does not protect a record. And the authors find that larger models were more vulnerable than smaller ones, so scale does not make the problem go away. Input: a prompt, possibly empty. Output: text that may be a memorized training example. A second attack asks a narrower question: membership inference is an attack that tells whether a given record was in a model's training set, and Shokri et al. (2017) study it against models that only answer queries.
Step 2: A deletion request reaches the weights, not just the database
A deletion request must be applied to every place the data lives, and the trained weights are the hard one. GDPR Article 17 gives a person the right to obtain erasure of personal data from the controller without undue delay on listed grounds, such as the data having been unlawfully processed or no longer being necessary for its purpose. It is not unlimited: the article also lists exceptions, for example the exercise of freedom of expression and information.
A defensible engineering answer has four parts. First, delete the data from raw stores, logs and the retrieval index. Second, exclude it from every future training set. Third, retrain or fine-tune on a schedule without it, and treat approximate unlearning methods, which try to remove one example's influence without a full retrain, as research to evaluate before relying on them. Fourth, add output filters and leakage tests so the model does not regurgitate it meanwhile. Whether weights count as personal data is a contested legal question; say so and involve counsel.
Step 3: Differential privacy adds calibrated noise
Differential privacy limits what any one person's data can change in a result. The National Institute of Standards and Technology (NIST) describes it as a mathematical framework that quantifies privacy loss to people whose data appears in a dataset. In its guidelines (Special Publication 800-226, final in March 2025), NIST notes that it is generally achieved by adding random noise to analysis results, and that more noise gives better privacy but lower utility.
The size of the protection is set by the privacy budget, called ε (epsilon): a smaller ε means each person's data matters less to the output and more noise is added. Note what differential privacy (DP) is not. It does not delete or anonymize records; it is a property of the algorithm that produces the answer. Each extra answer released spends more budget, because under basic composition the costs of separate queries add up. Input: a true result and ε. Output: a noisy result.
Step 4: Private training clips and blurs updates, or keeps data local
Private training applies the noise to the learning process itself. Differentially private stochastic gradient descent (DP-stochastic gradient descent (SGD)), introduced by Abadi et al. (2016), clips each example's gradient to a maximum norm C, so no one example can pull the weights far, then adds Gaussian noise with standard deviation σC to the summed gradient before the weight update. A tool the paper calls the moments accountant tracks the total privacy spent across all steps. Gaussian noise gives (ε, δ)-differential privacy, a slightly relaxed form in which δ is a small probability that the ε bound does not hold; the worked example below uses pure ε only.
Federated learning attacks a different part of the problem: raw data never leaves the device, and only updates travel. An update is computed from private data and can still carry information about it: Zhu et al. (2019) reconstruct training inputs from shared gradients. Federated learning is therefore normally combined with differential privacy and with secure aggregation, which lets the server see only the sum of many updates. Its practical limits are communication cost, uneven data across devices and harder debugging.
Step-by-step description
- Models can reveal training examples.
- A deletion request must reach the model.
- Differential privacy blurs each result.
- Clip and blur updates, or keep data local.
- A model can memorize training data.
- A deletion request must reach the model.
- Differential privacy adds noise to results.
- Private training clips and blurs updates.
Central versus local differential privacy
Central differential privacy trusts one curator who holds the raw data and adds noise to the results it releases. Local differential privacy is the variant in which each device adds noise to its own data before anything leaves it, so even the collecting server never sees a true value. The price is more noise per person, so local DP needs many users before the totals are useful.
Apple's research page Learning with Privacy at Scale (December 2017) describes local DP used to learn from many devices for features such as emoji suggestions and new words. Candidates report DP questions that ask for a company example like this one. Say which model you mean: central DP protects a released result, local DP protects the data at its source.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take the simplest private release: a count of users, with one user able to change the count by at most one. Adding Laplace noise with scale 1 divided by ε is the standard way to make this count differentially private, and the average size of the noise then equals 1 divided by ε.1 At ε = 0.1 the average error is 10, at ε = 1 it is 1, and at ε = 10 it is 0.1.1 The chance that the error exceeds 5 is about 0.61 at ε = 0.1 and about 0.007 at ε = 1.1
Now see why small groups are hard. For a total count of 1,000 records (illustrative), an average error of 1 is 0.1% of the count. For a subgroup of 20 records (illustrative) the same error is 5%, and at ε = 0.1 the error of 10 is 50% of the subgroup.1 The noise is the same size for everyone, so a small group loses more accuracy.
| Fact | Value |
|---|---|
| Average error of a count at ε = 0.1 | 10 records |
| Average error of a count at ε = 1 | 1 record |
| Average error of a count at ε = 10 | 0.1 records |
| Error at ε = 1 as a share of a group of 1,000 (illustrative) | 0.1% |
| Error at ε = 1 as a share of a group of 20 (illustrative) | 5% |
| Error at ε = 0.1 as a share of a group of 20 (illustrative) | 50% |
| Error at ε = 10 as a share of a group of 20 (illustrative) | 0.5% |
Composition adds a second cost. With a total budget of 1, one query gets ε = 1 and an average error of 1; split the same budget across ten queries and each gets ε = 0.1, so each carries an average error of 10.1
Text description of the diagram
- At epsilon 0.1 the average error is 10.
- At epsilon 10 the average error is 0.1.
Trade-offs and edge cases
Every control buys protection at a price, and none covers all three places data lives. The table below puts the five defenses side by side.
Text description of the diagram
- Deleting stored data does not change trained weights.
- Output filters miss paraphrased leaks.
- Differential privacy costs accuracy.
- Retraining costs compute.
Five edge cases deserve a sentence each. First, public data is not risk-free: the leaked items in Carlini et al. (2021) were scraped from the public internet. Second, DP is a trade, not a switch: as NIST states, the privacy-utility tradeoff can make high utility and strong privacy hard to get together. Third, output filters catch verbatim strings but miss paraphrases of the same fact. Fourth, evaluation data is user data too. Fifth, retraining costs compute, and every scheduled retrain spends it again.
For the question of sourcing real production traffic for evaluations, the following is engineering practice, not a legal standard. Redact PII before storage, sample the minimum needed, set a retention limit, restrict access, and prefer synthetic or paraphrased cases where they test the same behavior. The same care applies when a judge model reads the text, as covered in LLM-as-a-judge and in retrieval-augmented generation (RAG) evaluation with Ragas. Attack-side risks such as prompt injection are in AI agent security.
How to answer it in an interview
A user invokes the right to be forgotten, but their data is in your model weights. How do you comply?
I would treat it as four layers. First, delete their data from raw stores, logs and the retrieval index. Second, exclude it from all future training sets. Third, retrain or fine-tune on a schedule without it, and evaluate approximate unlearning before I trust it. Fourth, run leakage tests and output filters so the model cannot repeat it in the meantime. The limit is that weights are not a table, so I would tell legal exactly what the model still contains and when the next retrain removes it.
The spoken structure: name the places data lives, give the four steps in order, then state the limit honestly.
Common mistakes
- "Differential privacy anonymizes the data." (wrong) It does not remove anything. (right) "Differential privacy limits how much one person's data can change a released result, by adding noise."
- "Rare records are safe because they appear once." (wrong) Single-document sequences were extracted in Carlini et al. (2021). (right) "Rarity does not protect a record; I test for memorization."
- "Federated learning is private by itself." (wrong) Updates come from private data. (right) "I combine it with differential privacy and secure aggregation."
- "We deleted the row, so the request is done." (wrong) The weights still carry its influence. (right) "I delete, exclude, retrain on a schedule and filter outputs."
- "A bigger model is safer." (wrong) Larger models were more vulnerable in the same study. (right) "Scale raises the need for leakage tests."
Follow-up questions
If you discovered that your model was exposing sensitive data from its training set, what would you do next?
Contain it first: add an output filter or restrict the affected feature, and preserve the prompts that reproduce the leak. Then trace the source in the training data, remove it, plan a retrain, and add that leak as a permanent regression test.
How does federated learning work and what are its limitations?
Devices train locally and send updates that the server averages into one model. The limits are communication cost, uneven data and the fact that updates can still reveal information, so it is paired with differential privacy.
What if retraining without the person's data is too expensive?
Schedule retrains so each deletion is honored by a known date, and block the data from outputs with filters until then. Treat approximate unlearning as an option only after you have measured that it removes the influence.
How do you prevent sensitive enterprise data from leaking into public models?
Keep it out of training and prompts by default: redact before sending, use contracts and settings that limit vendor retention, and check them in the vendor's documentation. Test with leakage probes before launch.
How would you design an implementation for a customer in a regulated industry?
Reported by candidates as a design question; the following is engineering practice. Agree a data processing agreement, set retention to the minimum, pick the region where data is processed, redact sensitive fields before sending, keep audit logs, and verify each setting in the vendor's documentation.
How do you source real production traffic for evals without breaking privacy?
Redact PII, sample the minimum, limit retention and access, and replace real cases with synthetic ones where the test allows. Document the basis for using the data. The LLM evaluation guide shows where such sets fit.
- A model can memorize and reveal training examples, even ones seen once, and larger models were more vulnerable.
- A deletion request needs four steps: delete stored copies, exclude from training, retrain on a schedule, filter outputs.
- Differential privacy adds calibrated noise; a smaller privacy budget means more noise and less accuracy.
- DP-SGD clips and blurs gradients; federated learning keeps raw data local but needs more protection.
- Test for leaks, because an untested control is only a hope.
Sources
- Extracting Training Data from Large Language Models (arXiv 2012.07805) — arXiv (Carlini et al., USENIX Security 2021), 15 Jun 2021
- NIST SP 800-226: Guidelines for Evaluating Differential Privacy — NIST, 6 Mar 2025
- GDPR Article 17: Right to erasure ('right to be forgotten') — gdpr-info.eu (text of Regulation 2016/679), 25 May 2018
- Deep Learning with Differential Privacy (arXiv 1607.00133) — arXiv (Abadi et al., ACM CCS 2016), 1 Jul 2016
- Learning with Privacy at Scale — Apple Machine Learning Research, 6 Dec 2017
- Membership Inference Attacks against Machine Learning Models (arXiv 1610.05820) — arXiv (Shokri et al., IEEE S&P 2017), 18 Oct 2016
- Deep Leakage from Gradients (arXiv 1906.08935) — arXiv (Zhu et al., NeurIPS 2019), 21 Jun 2019
- NIST SP 800-226: Guidelines for Evaluating Differential Privacy (PDF) — NIST, 6 Mar 2025
