In this article
Before you start
This page is part of the machine learning system design guide, and the statistics are covered in more depth on the time-series anomaly detection page.
Why interviewers ask this
Anomaly detection looks like a modelling question and is mostly a systems and product question. The interviewer wants to see that you can say what an anomaly is for this product, who acts on an alert, what a false alarm costs, and how the design copes with a high-volume stream. It appears in many forms: network traffic, payments, telemetry and position tracks.
A weak answer picks an algorithm first. A strong answer starts with the cost of errors and the baseline, picks the simplest scoring that works, and spends most time on alert volume and the streaming path. Related: distribution shift, since a baseline ages, and batch, streaming and real-time inference.
Designing an anomaly-detection system from first principles
In plain words
- Anomaly
- An anomaly is a reading, or a run of readings, that departs from the expected pattern for that metric at that time by enough that a person or a process should look at it.
- Alert fatigue
- Alert fatigue is the habit of ignoring alerts that grows when most of them are false or call for no action, so that a real one is missed.
Think of a night guard who knows the building. At 3 a.m. the lift moving is odd; at 9 a.m. it is not, and not every creak means a call. Each metric has its own normal. The analogy breaks in one place: the system must learn each metric's normal from stored history, and a new service may have only a few hours.
In one sentence, the system learns a baseline per series, measures each reading's distance from it, and alerts only when that distance is large and lasts. A z-score is that distance: how many standard deviations a reading sits from its expected value.
Step 1: Define the anomaly and the cost of errors
The first step writes down what counts as an anomaly and who acts on it. The input is the product question, such as "page an engineer when traffic collapses"; the output is an anomaly definition, an owner and a ranking of errors. A missed event and a false alarm cost different amounts in different products, and that ratio sets every threshold later.
Then ask whether labels exist. The scikit-learn guide separates outlier detection, where the training data contains outliers and the model fits the dense region, from novelty detection, where the training data is clean (scikit-learn 1.9.1). Google's site reliability engineering (SRE) book adds a rule for the output: every page, meaning an alert that wakes a person, should be actionable (Google SRE book, chapter 6). What you now know: the definition, the owner and the error costs set the design.
Step 2: Build a baseline that knows the season
The second step builds the expected value for each series and hour. The input is history; the output is a baseline and a measure of normal spread. Most operational metrics have a daily cycle, so one global average is wrong most of the time. In the worked example below, a metric swings about 20 up and down over a day, so its overall standard deviation is 14.46 and three standard deviations is about 43 away from the mean. Anomalies of 10 and 14 never reach it, and a global rule catches 0 of 8.1
Fit the cycle instead. A least-squares fit of a daily sine and cosine to three days of history leaves residuals with a typical spread of 3.03, so three spreads is about 9, and all 8 events are caught.1 The spread is measured with the median absolute deviation (2.05 here), which one odd reading barely moves, times 1.4826 to match a standard deviation for normal noise, giving 3.03. Three spreads is 3 x 3.03 = 9.1. What you now know: remove the season before measuring distance.
Step 3: Score each point or window
The third step turns each new reading, or each short window of readings, into one score. The input is the reading and its baseline; the output is a number where larger means stranger. For one metric, the z-score above is enough. For many related metrics, a model scores several at once. An isolation forest isolates a point with random splits and treats short average path lengths as anomalous; its contamination setting sets the score threshold so that about that share of the training points is flagged (scikit-learn IsolationForest, 1.9.1).
A forecasting model scores the gap between its prediction and the reading. Hundman et al. used long short-term memory (LSTM) networks on spacecraft telemetry with a nonparametric dynamic threshold and techniques to cut false positives (Hundman et al., 2018). Seasonal statistics are cheap and explainable, so start with the baseline and add a model only where it falls short. What you now know: a score is a distance from normal, however it is computed.
Step 4: Turn scores into few, useful alerts
A score is not an alert. The input is a stream of scores; the output is a small number of alerts that deserve attention. Scoring ten thousand samples a second means 864 million decisions a day, and a false-alarm rate of 0.1 percent on each one gives 10 false alarms every second, or 864,000 a day. Making one decision per one-minute window for one metric gives 1,440 decisions a day, and the same rate gives about 1.4 false alarms a day.1
Add persistence: alert only when 3 of the last 5 minutes are flagged. In the toy run, this cut false alerts from 5 to 0, and also dropped the four one-minute spikes, so the answer is two tiers: page on persistent deviations, and file a ticket for brief spikes. Prometheus advises to "allow for slack in alerting to accommodate small blips" and to link each alert to a dashboard (Prometheus). What you now know: alert volume is a design target, not a by-product.
Step 5: Run it as a stream that keeps learning
The fifth step runs the pipeline on live data and keeps it healthy. The input is the event stream; the output is alerts plus a feedback loop. Partition the stream by series key so each worker keeps the rolling window and baseline for its series in memory, and snapshot that state so a restart does not lose it. Run a slower second-stage model only on windows the cheap check flags.
Refit baselines on a schedule, and exclude flagged points so the baseline does not learn the anomaly as normal. Record what responders do with each alert, because those labels turn later tuning from guesswork into measurement. Monitor the detector itself: Prometheus's guidance includes alerting when the monitoring system is down (Prometheus). What you now know: state, a cheap first stage, a refit schedule and feedback complete the design. The score is the third step of five, and the alert rule in step four decides whether anyone reads it.
Step-by-step description
- Name the anomaly and the cost of errors.
- Learn what normal is for each hour.
- Measure distance from the baseline.
- Page only on persistent, actionable scores.
- Keep state, learn, take feedback.
- The score is the third step of five.
- The alert rule in step four decides whether anyone reads it.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The example uses a synthetic metric invented for teaching: seven days of one reading a minute, a daily cycle of 100 plus or minus 20, noise with a spread of 3, and eight injected anomalies in the last four days. Four are one-minute spikes of 14 and four are 15-minute shifts of 10. The first three days train the detectors; the script is deterministic with seed 0.1
| Detector (four evaluation days) | Events caught of 8 | Alerts | False alerts | False alerts per day |
|---|---|---|---|---|
| Global z-score, 3 spreads | 0 | 0 | 0 | 0.00 |
| Seasonal baseline, 3 spreads | 8 | 13 | 5 | 1.25 |
| Seasonal plus 3-of-5 persistence | 4 | 4 | 0 | 0.00 |
The persistence rule caught the four 15-minute shifts and missed the four one-minute spikes.1 Which one to deploy depends on Step 1: if a spike is a real incident, keep a separate single-minute rule with a higher threshold.
Text description of the diagram
- A global z-score caught 0 of 8 events, a seasonal baseline caught 8 with 5 false alerts, and adding persistence caught 4 with 0 false alerts.
- Persistence removed every false alert but also dropped the four one-minute spikes.
| Quantity (illustrative false-alarm rate of 0.001) | Value |
|---|---|
| Samples per second | 10,000 |
| Samples per day | 864,000,000 |
| False alarms per second if every sample is a decision | 10 |
| False alarms per day, every sample | 864,000 |
| Decisions per day at one per minute | 1,440 |
| False alarms per day, one per minute | 1.44 |
Trade-offs and edge cases
The design trades detection speed against false alarms, and model power against explainability.
- Threshold and window. A lower threshold or shorter window finds events sooner and raises more false alarms. Tune on the cost ratio from Step 1, not on a default.
- Baseline drift. A baseline fitted once goes stale when traffic grows or a release changes behaviour. Refit on a schedule and watch for distribution shift.
- Holidays and launches. A cycle fitted on ordinary weeks flags a holiday as anomalous. Add calendar features or suppress alerts for known events.
- New series. A series with a few hours of history has no season. Borrow the baseline of similar series or use a wider threshold until history grows.
- Learning the anomaly. If an outage is included in the next fit, the baseline accepts it. Exclude flagged points from refits.
Google's SRE guidance says to page on symptoms such as user-visible errors and to alert on causes only when they are very definite and imminent (Google SRE book, chapter 6), while a detector watching internal metrics is a cause-level tool; use it to explain a symptom, and page from the symptom. The same chapter says Google's teams "avoid 'magic' systems that try to learn thresholds", keeping paging rules simple and keeping a very quick rule for sudden drops in user request rates; that is a reason to keep the paging rule on top of a detector simple and explainable.
Text description of the diagram
- Seasonal statistics are cheap and explainable.
- Add a model only where it falls short.
How to answer it in an interview
Design an anomaly-detection system.
I would start with what an anomaly is for this product, who acts on an alert, and what a false alarm costs. For each metric I would fit a seasonal baseline from its history and score each window by its distance from that baseline in units of typical spread. The key is alert volume: I would decide per minute, not per sample, require the deviation to persist, and page on persistent ones while ticketing brief spikes. It runs as a stream partitioned by series with state in memory, a refit schedule that excludes flagged points, and responder feedback.
The spoken structure is: the cost of errors, the baseline, the score, the alert rule, then the stream and feedback.
Common mistakes
- Choosing an algorithm first. (wrong) "I would use an autoencoder." It is wrong because the right model depends on what an error costs, which you have not yet defined. (right) Define the anomaly, the owner and the cost of errors, then pick the simplest score that works.
- One global threshold. (wrong) "Flag anything three standard deviations from the mean." (right) Remove the daily and weekly cycle first; in the example, the global rule caught none of the eight events.
- Alerting on every score. (wrong) "Each flagged reading raises an alert." It is wrong because noise flags readings constantly and alert fatigue follows. (right) Aggregate to windows and require persistence.
- Learning the outage. (wrong) "Retrain on all the data each night." (right) Exclude flagged points so the baseline does not absorb the anomaly.
- No feedback loop. (wrong) "Ship it and tune later." It is wrong because without responder labels you cannot tell a good threshold from a bad one. (right) Record what responders did with each alert and use it to tune.
Follow-up questions
How would you identify ships that deviate from their course?
Build a baseline of normal tracks per route, such as expected lanes, speeds and headings, and score each new position by its distance from the expected path. Add a per-vessel check against the ship's own history, and alert on a deviation that persists across several position reports.
How would you detect anomalies in network traffic at very large scale?
Aggregate to counters per prefix, port or service per time window, keep a seasonal baseline per segment, and score windows, not packets. Use a cheap statistical first stage on all segments and a heavier model only on flagged ones.
How do you balance algorithmic accuracy against system latency?
Use two stages: a fast statistical check on every window, and a slower model on only the flagged windows. Set the window length by how quickly someone must act; shorter windows are faster and noisier.
How would you design a transaction anomaly detection system?
Compute features per account, such as amount against its usual range and recent velocity, score with rules plus a model, and size the review queue to what analysts can handle. Labels arrive late from confirmed cases, so plan for delayed feedback.
What changes when you have labelled anomalies?
You can train a supervised classifier and measure precision and recall directly, but labelled anomalies are rare and may miss new kinds, so keep the unsupervised baseline as well. See the confusion matrix page for the metrics.
- Define the anomaly, the owner and the cost of a false alarm before choosing an algorithm.
- Remove the season first: a global z-score caught 0 of 8 events in the example and a seasonal baseline caught all 8.
- A score is not an alert: decide per window and require persistence, and ticket brief spikes.
- Treat alert volume as a design target: 864,000 false alarms a day when every sample is a decision, against 1.44 a day with one decision a minute.
- Run it as a partitioned stream with refits that exclude flagged points, and keep responder feedback.
Sources
- Monitoring Distributed Systems (Google SRE book, chapter 6) — Google, Ewaschuk (copyright 2017), 1 Jan 2017
- Alerting (Prometheus practices) — Prometheus (page undated; read 2026-10-24), 24 Oct 2026
- Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding — arXiv (Hundman et al.; v1 February 2018), 13 Feb 2018
- Novelty and Outlier Detection (scikit-learn user guide 1.9.1) — scikit-learn (version 1.9.1 documentation; read 2026-10-24), 24 Oct 2026
- IsolationForest (scikit-learn 1.9.1 API reference) — scikit-learn (version 1.9.1 documentation; read 2026-10-24), 24 Oct 2026
