Adversaries adapt: what it means for ML system design

In this article

1. Before you start
2. Why interviewers ask this
3. Adversaries adapt from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the machine learning (ML) system design interview guide, and the full fraud pipeline is on the real-time fraud detection page.

Why interviewers ask this

The question separates candidates who treat a fraud model as a one-off training job from candidates who treat it as a contest that never ends. Candidates report prompts about bot players, fraudulent accounts, scam posts, device graphs and delayed fraud labels (reported by candidates, with no public source for which company asks which). Every one of them hides the same trap: the usual machine learning (ML) habit of training once and watching accuracy does not hold when someone is working against you.

A weak answer lists features and a model type. A strong answer says who the adversary is, what they can see, how fast they react and how the system keeps working while labels arrive late. A Staff-level answer also states what cannot be fixed and what the design does about it.

Adversaries adapt from first principles

Adversarial drift

Adversarial drift is the change in the incoming data that happens because an actor reacts to a deployed model, shifting behaviour to get bad items past it. It differs from ordinary drift because the cause is the model itself, so the shift is a response to your own decisions.

Label delay

Label delay is the gap between the time a model scores an item and the time its true outcome becomes known, such as a chargeback that arrives weeks after a payment. During the gap, the newest data has no label or a biased one.

Evasion attack

An evasion attack is an attempt to change an input at prediction time so that a deployed model labels a bad item as good, without changing the model itself.

In plain words

Think of airport security. If screeners always look for the same bag shape, smugglers quickly learn to pack differently, and the check that worked last month passes more bags each week. The analogy breaks in one place: a screener sees most bags, while a model often learns about its misses only when a label arrives, sometimes weeks later.

In one precise sentence, the data-generating process of a fraud model is not fixed, because the adversary is part of it. Dalvi et al. (2004) state that nearly all data mining algorithms assume the data-generating process is independent of the data miner's activities. They add that in adversarial domains the performance of a classifier "can degrade rapidly after it is deployed".

Step 1: The model goes live and the attacker starts probing

A deployed model becomes a target as soon as it blocks something. The attacker probes the deployed model. Each blocked attempt tells the attacker that something in the attempt was wrong, and each accepted attempt tells them it was fine. The attacker does not need the model's weights: Lowd and Meek (2005) note that earlier work wrongly assumed the attacker knows the classifier perfectly, and introduce the adversarial classifier reverse engineering (ACRE) problem, which is learning enough about a classifier to build attacks.

Input: the live model's decisions. Output: the attacker's knowledge of what gets through. You now know why even a hidden model leaks: its answers are the leak. The next step is what the attacker does with that knowledge.

Step 2: The attacker finds and spreads an evasion

The attacker changes the inputs the model uses, so that bad items look like good ones. Lowd and Meek describe spammers disguising messages by adding words that are more typical of legitimate email than of spam. The adversary's goal, in the words of Dalvi et al., is to make the classifier produce false negatives.

Once one evasion works, it spreads, because attackers share methods and scripts across accounts or devices. Once a trick works, it can be reused across many attempts within days. Input: probing results. Output: a changed input pattern that scores as good. The model has not changed, but the data has, and that mismatch is the start of adversarial drift.

Step 3: Recall decays while the dashboard still looks calm

The model keeps scoring, but fewer of the real bad items are caught. Dashboards that show precision on blocked items can look healthy, because the missed items are exactly the ones nobody reviewed. Only recall, which needs labels for the items that got through, shows the damage, and those labels are slow.

In the worked example below, an illustrative detector that starts at a recall of 0.9 and loses 15% of it each week falls to a recall of 0.3394 after six weeks.¹ The key point: the loss does not show in any metric that only counts blocked items. Input: shifted data. Output: a quiet fall in recall. The fix people reach for next is retraining, and Step 4 shows why that alone trails.

Step 4: Retraining on late labels trails the attacker by the label delay

Retraining teaches the model the attacks that were in the labelled data, and the labelled data is old. If fraud outcomes take 21 days to arrive, the newest three weeks of training data are unlabelled or biased, so each new model describes the attacker as they were three weeks ago.

Dalvi et al. (2004) call the standard remedy, repeated manual reconstruction of the classifier, the only solution then available, and propose treating classification as a game in which the classifier is optimal against the adversary's best response. In the example, weekly retraining with a 21-day label delay ships a detector with recall 0.5527, against 0.9 with no delay, and recall keeps falling until the next retrain.¹ Input: delayed labels. Output: a model that is about three weeks stale when it ships. The loop then repeats from Step 1.

1 · Model goes live2 · Evasion found3 · Recall decays4 · Late retrainReplay

Step-by-step description

1. Every decision tells the attacker something.
2. The attacker shifts inputs to score as good.
3. Missed items are not reviewed, so it stays quiet.
4. Labels are weeks old, so the model lags.

• The attacker probes the deployed model.
• An evasion spreads and recall decays.
• Retraining on late labels only learns the old evasion.

Worked example

Take an illustrative detector with a starting recall of 0.9. Assume each week attackers move and the detector keeps 85% of its recall (both numbers are illustrative, not measurements). Recall then falls to 0.765 after one week, 0.5527 after three weeks and 0.3394 after six.¹

Now add retraining. A retrained model is only as fresh as its newest labels, so a label delay of 7, 21 or 42 days leaves it stale by one, three or six weeks. Out of 1,000 illustrative fraud attempts a week, a detector at those recall levels misses 100 with no delay, 235 with a 7-day delay, 447 with a 21-day delay and 661 with a 42-day delay.¹ Recall when each retrain ships is 0.9 with no label delay and 0.3394 at 42 days; it then keeps falling for a week until the next retrain, so these are best-case values.

All values are illustrative and computed.¹ A second computed quantity: if a wrongly blocked good customer costs 1 unit and a missed fraud costs 50 units (illustrative), the cost-minimising rule is to block when the fraud probability exceeds 1 divided by 51, which is 0.0196.¹ The threshold is a business decision, and an attacker who learns it can aim just under it.

Text description of the diagram

• Recall when each retrain ships is 0.9 with no label delay.
• Recall when each retrain ships is 0.3394 at 42 days.

Trade-offs and edge cases

Adversarial drift is a special case of drift. Concept drift is a change over time in the relationship between inputs and labels, whatever its cause. Natural concept drift (a seasonal shift in purchases) happens whether or not a model is deployed. Adversarial drift is drift caused by the model's own decisions: it targets what the model currently misses, so it moves faster where the model is strongest and a fixed retraining schedule trails it.

Text description of the diagram

• Natural concept drift happens whether or not a model is deployed.
• Adversarial drift is drift caused by the model's own decisions.

Evasion is not the only attack. The National Institute of Standards and Technology (NIST) taxonomy of adversarial machine learning (March 2025) organises attacks by ML method, life-cycle stage and the attacker's goals, capabilities and knowledge. In the report itself, for predictive AI it groups attacks into evasion, poisoning (data and model poisoning) and privacy attacks, against the objectives of availability, integrity and privacy. Data poisoning is an attack that corrupts the training data so that the learned model behaves wrongly, which matters when your training labels come from user reports an attacker can fill with noise.

Practical edge cases a strong answer names:

• Faster signals help: use features that are costly for the attacker to change, such as links between accounts, and review samples of items the model allowed to catch false negatives.
• Randomness helps: a decision threshold that the attacker cannot map exactly is harder to aim under.
• Experiments get harder: if attackers act on both the control and the treatment model, the two groups are no longer independent, so state that limit.

Why the model's own blocks bias its labels

Training labels are censored by the model's own decisions: an item the model blocks never produces an outcome, because a declined payment cannot charge back and a removed post cannot later be reported as missed. The next training set then holds mostly items the old model allowed, so it learns little about the attacks the old model already stops.

The standard response is a small random holdout that lets a sample of would-be-blocked traffic through at a capped cost, or human review of a sample of blocked items. A second problem is label noise. Telling a scam from a post that only looks scammy needs expert-reviewed labels, written policy definitions and agreement checks between reviewers, because user reports alone mix real scams with items users merely dislike.

A broader view of spotting unusual behaviour is in designing an anomaly-detection system and the abuse side in harmful-content detection. Credit-risk models also learn from outcomes that arrive late; see designing churn, customer value over time (LTV) and credit-risk prediction.

How to answer it in an interview

The spoken structure: say the attacker reacts, name the measurement problem, then the retraining lag, then the trade-off.

Common mistakes

• "I train once and monitor accuracy." (wrong) The attacker changes the data after launch, and accuracy on blocked items hides the misses. (right) "I measure recall on a reviewed sample of allowed items and retrain on a schedule."
• "The attacker needs the exact model." (wrong) Querying a model can reveal enough to build evasions, as Lowd and Meek showed for linear classifiers. (right) "I assume the attacker learns from my decisions."
• "Weekly retraining solves it." (wrong) The newest labels are still late, so the model is stale by the label delay. (right) "Retraining reduces the lag, and fast rules and hard-to-fake features cover the gap."
• "Adversarial machine learning means adversarial images." (wrong) The classic setting is spam, fraud and intrusion detection. (right) "Here the adversary is a person adapting to my decisions, and image attacks are one case."

Follow-up questions

How do you handle feedback delay when fraud labels arrive weeks late?

Train on mature cohorts only, use early proxy signals such as refunds or account reports, and review a sample of recent allowed items by hand. Add fast rules for new patterns while the learned model catches up.

How would you use login information and device activity to build a device graph?

Link accounts that share devices, addresses or login patterns into a graph, then score clusters as well as single accounts. Cluster features are costly for an attacker to fake, but shared devices also join innocent users, so keep the evidence weighted.

How do you design a system to detect bot players?

Combine behaviour features, such as input timing and session patterns, with device and account links, and expect the bots to change once a rule blocks them. The fraud flow is covered in real-time fraud detection.

How do you train a model to tell a scam from a post that only looks scammy?

Use expert-reviewed labels with written policy definitions and checks that reviewers agree, rather than user reports alone. Keep a small random holdout of would-be-blocked items so the labels are not censored by the model's own blocks.

What if labels arrive in two days instead of 21 days?

The retrained model then ships less than a week stale, so weekly retraining leaves recall close to the no-delay case. The attacker's own pace becomes the limit, so retrain more often and keep the reviewed sample of allowed items.

How do you design a real-time system to detect fraudulent accounts or spam, addressing adversarial drift and latency constraints?

Use a cheap fast model on the request path with a slower model and graph checks running behind it, and feed the late labels back into retraining. Monitor recall on a reviewed sample so drift shows up before the loss does.

Sources

1. Adversarial classification (Dalvi, Domingos, Mausam, Sanghai, Verma) — ACM SIGKDD 2004, 22 Aug 2004
2. Adversarial Learning (Lowd and Meek) — ACM SIGKDD 2005, 21 Aug 2005
3. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025) — National Institute of Standards and Technology, 24 Mar 2025
4. NIST AI 100-2e2025 report (PDF) — National Institute of Standards and Technology, 24 Mar 2025