In this article
Before you start
This page is one step of the machine learning (ML) system design interview learning path.
Why interviewers ask this
A classification prompt looks small, but it hides every production question: what to optimise, what to do when the model is unsure, how to keep latency low, and how to notice when the model decays. Candidates report variants such as classifying documents for enterprise customers, routing bug reports to teams, analysing request-for-proposal (RFP) documents with generative AI, and predicting the language of social posts.
A weak answer names one model ("fine-tune a transformer") and one metric ("accuracy"). A strong answer starts from the label set and the cost of each error, then builds a pipeline with a cheap first stage. A Staff-level answer adds the feedback loop: low-confidence documents reach people, and their decisions become the next training set. The same loop appears in harmful-content detection.
Design a text-classification system at scale from first principles
- Text-classification system
- A text-classification system is a pipeline that reads each incoming document, assigns it labels from a fixed set with a confidence score, and returns them within a latency budget, while collecting feedback so the model can be retrained when the data changes.
- Feature hashing
- Feature hashing is a way to turn words into numbers that maps each token to a column index with a hash function instead of a stored vocabulary, so the feature space has a fixed size and needs no fitting step, at the price of occasional collisions.
In plain words
Think of a mailroom. Most letters carry a clear address and a clerk sorts them in a second. A few are smudged. A good mailroom does not make the clerk guess; the smudged letters go to a supervisor. The analogy breaks in one place. A mailroom clerk knows when a letter is smudged. A model must be taught to report how unsure it is, and its confidence score has to be checked against real outcomes.
In one precise sentence, a text-classification system turns text into features, scores each label with a model, accepts the label only when the confidence clears a threshold, and sends the rest to a costlier path.
Step 1: Fix the labels, the metric and the budget
Start by asking what a label means and what a wrong label costs. Decide whether each document gets one label or several, and whether new labels will appear later. Then choose the metric, because the wrong one hides failure. Accuracy is the share of all predictions that are correct, and on rare positives it misleads: Google's crash course notes that a model predicting negative every time on data with 1% positives scores 99% accuracy and is useless.
Precision is the share of predicted positives that are right. Recall is the share of real positives that were found. The F1 score is the harmonic mean of the two. For rare classes the crash course on receiver operating characteristic (ROC) and area under the curve (AUC) says precision-recall curves may compare models better than the receiver operating characteristic (ROC) curve and its area under the curve (AUC). Finally, write the budget: documents per day, peak rate, latency target. You now have a target; the next step makes the first model cheap.
Step 2: Turn text into features and score it with a cheap model
The first model should be the cheapest one that is good enough, because it sees every document. A linear classifier on word and phrase counts is a credible baseline: the fastText paper reports it is often on par with deep classifiers in accuracy and many orders of magnitude faster, training on over a billion words in under ten minutes on a multicore central processing unit (CPU).
To build the features at scale, use feature hashing. The scikit-learn HashingVectorizer is stateless, so it needs no stored vocabulary and no fit call, and it suggests 2 ** 18 features for text classification. The costs are collisions, where distinct tokens share a column, and no inverse document frequency (IDF) weighting. The feature extraction guide describes streaming mini-batches so memory stays bounded by one batch. Output of this step: label scores and a confidence for each document.
Step 3: Low-confidence documents go to a stronger model or a person
Not every document deserves the same compute. A cascade is a chain of models ordered from cheap to costly, where each stage keeps the documents it is confident about and passes the rest on. Set a confidence threshold on held-out data, picking the point where the accepted documents reach your target precision; documents below it are the system's abstentions. Raising the threshold lifts the abstention share, which costs more stronger-model and review time, and cuts accepted errors. The pass-on rate p sets the mean cost: 20 + p × 150 milliseconds in the worked example below.
The next stage is usually a pretrained transformer adapted to the task, a step called fine-tuning (training it further on your labelled examples). A smaller one often suffices: Sanh et al. (2019) report that DistilBERT, a distilled version of the Bidirectional Encoder Representations from Transformers (BERT) model, is 40% smaller and 60% faster and keeps 97% of its language-understanding ability. The final stage is a human review queue. Its decisions are the most valuable data you collect, since they are exactly the cases the model finds hard. Output: a label, or a ticket for a person.
Step 4: Serve two lanes, interactive and bulk
Interactive requests (a user uploads a file and waits) and bulk work (relabelling a backlog) have different goals. Interactive traffic needs low latency at the 99th percentile, so keep workers stateless, cap queue length and keep a little spare capacity. Bulk work needs throughput, so batch documents and run it on separate workers so it never starves the interactive lane.
Cheap tricks add up: cache results by a hash of the normalised text, because enterprise documents repeat; do feature extraction in the same code for training and serving, which stateless hashing makes easy; and store each result with the model version that produced it so a retrain can be compared. The worked example below sizes both lanes with illustrative numbers. You now have the serving shape; the last step keeps it correct over time.
Step 5: Monitor, collect labels and retrain
A deployed classifier decays when the data changes. Track three things: per-class precision and recall on a small random sample that people label every week; the share of documents that abstain, because a rising share is an early warning that inputs have moved; and the mix of predicted labels against the past. Alerts on these tell you when to retrain. Monitoring and retraining close the loop.
Reviewed labels retrain the cheap model, together with a fresh random sample, so the training set does not only contain hard cases. Compare the new model against the old on the same held-back set before switching, and ship it to a slice of traffic first. This is why pre-deployment numbers can mislead; see why pre-deployment metrics can look great while the live test does not. Where labels are very rare, the problem begins to resemble anomaly detection.
Step-by-step description
- Fix labels, metric and budget.
- Hashed features, linear scores.
- Unsure documents go upward.
- Interactive and bulk serving.
- Reviewed labels retrain.
- Requirements fix the labels, the metric and the budget.
- A cheap model scores every document.
- Low-confidence documents go to a stronger model or a person.
- Serving uses an interactive lane and a bulk lane.
- Reviewed labels retrain the cheap model.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Two small calculations show the metric trap and the capacity plan. The first uses the confusion matrix from Google's glossary: 30 true positives, 20 false negatives, 950 false positives and 4,999,000 true negatives. Recall is 30 / (30 + 20) = 60% and accuracy is 99.98%, as the glossary states. Precision is 30 / (30 + 950) = 3.06%, and F1 is 5.83%.1 Only 1 in 33 positive predictions is right, although accuracy looks perfect.
The second is a capacity sketch. The inputs are illustrative assumptions, not sourced facts: 500,000 new documents a day, a 10,000,000-document backlog, 20 milliseconds per document for the cheap model and 150 milliseconds for the stronger one, with 15% of documents passed on, and 8 workers batching 500 documents per second each for the cheap stage of the backlog. Batching is why the bulk rate is far above the 50 per second of an unbatched worker.
The stronger stage is sized too, because the 15% passed on still need it. The DistilBERT row uses the paper's measured timings of 410 and 668 seconds for a full pass of the General Language Understanding Evaluation (GLUE) Semantic Textual Similarity (STS) Benchmark (STS-B) on a CPU at batch size 1 (Sanh et al., 2019, Table 3), a benchmark and not a production latency.
| Quantity | Value |
|---|---|
| Accuracy, glossary example | 99.98% |
| Recall, glossary example | 60% |
| Precision, glossary example | 3.06% |
| F1, glossary example | 5.83% |
| Average load, 500,000 documents a day (illustrative) | 5.79 per second |
| Peak load at ten times the average (illustrative) | 57.87 per second |
| Interactive workers for the cheap stage, 50 per second each, half loaded (illustrative) | 3 |
| Stronger-stage load at peak, 15% passed on (illustrative) | 8.68 per second, 1.30 busy workers, 3 workers at half load |
| Backlog, cheap stage: 10,000,000 documents, 8 workers at 500 per second each (illustrative) | 2,500 seconds, 41.67 minutes |
| Backlog, stronger stage: 1,500,000 documents at 150 milliseconds (illustrative) | 225,000 worker-seconds, 62.5 worker-hours |
| Cascade mean latency, 15% passed on (illustrative) | 42.5 milliseconds |
| Latency of a passed-on document, which sets the 99th percentile (illustrative) | 170 milliseconds |
| Stronger model alone (illustrative) | 150 milliseconds |
| Weights for 2 ** 18 features, 20 classes, 4 bytes each (illustrative) | 20,971,520 bytes |
| Human review at 2% of documents, 200 per reviewer (illustrative) | 10,000 a day, 50 reviewers |
| DistilBERT time against BERT-base, same benchmark | 39% less (410 against 668 seconds) |
All values follow by arithmetic from the stated inputs.1 The paper's headline "60% faster" and the table's 39% less time use different bases: 668 / 410 is 1.63 times faster, while 410 / 668 is 0.61 of the time.1
Text description of the diagram
- Accuracy is 99.98% while precision is 3.06%.
- Recall is 60% and F1 is 5.83%.
Trade-offs and edge cases
The central trade-off is cost against quality at each stage. A linear model on hashed features is the cheapest and fastest to run; a distilled transformer often reads subtle text better than a linear model, at moderate cost; a prompted large language model (LLM) needs few labelled examples but costs the most per document and is the slowest. On subtle text the distilled and prompted options vary: with enough labels a smaller model tuned on them can match a prompted LLM. These are typical orderings from engineering practice, so measure them on your own data.
Text description of the diagram
- A linear model on hashed features is the cheapest and fastest to run.
- A prompted large language model needs few labelled examples but costs the most per document and is the slowest.
Edge cases a strong answer names:
- Collisions and inspectability. Hashing keeps memory fixed, but a column holds several tokens, and weights cannot be read back as words without a vocabulary.
- Class imbalance. Report precision and recall per class, and compare models with precision-recall (PR) curves, as Google advises.
- New labels. A fixed head needs retraining when a team or category appears; keep an "other" label and review it.
- Privacy. Documents may not leave a tenant or a device; see on-device and privacy-constrained ML.
- Adversarial senders. When someone tries to evade the labels, the problem looks like real-time fraud detection.
How to answer it in an interview
How would you design a text-classification system for millions of documents?
I would start by fixing the label set and the metric. With rare classes I would track precision and recall per class, not accuracy. The first model is a linear classifier on hashed n-gram features, because it is cheap and streams. Every prediction carries a confidence. Above a threshold it is final; below it goes to a distilled transformer, and the last slice goes to a human review queue whose decisions become training data. I would serve interactive and bulk traffic on separate lanes and monitor per-class precision and the abstention rate to know when to retrain. The threshold trades cost against error.
The spoken structure: say the labels and metric, then the cheap first stage, then the confidence gate and the human loop, then serving and monitoring, and finish on the threshold trade-off.
Common mistakes
- "I would report accuracy." (wrong) On rare positives a useless model can score very high. (right) "I would report precision, recall and F1 per class, and compare models with precision-recall curves."
- "I would fine-tune a large transformer on every document." (wrong) That pays the highest cost for the easy majority. (right) "A cheap model labels the confident cases and a stronger one handles the rest."
- "The model always returns a label." (wrong) A forced label on an unsure case is an error you cannot see. (right) "Below the threshold the system abstains and routes to a person."
- "I would build the vocabulary first, then train." (wrong) That adds a stateful step, and the model ignores new words until the vocabulary is rebuilt. (right) "Hashed features need no stored vocabulary, so new words do not change the feature space."
Follow-up questions
How would you design a document classification pipeline for millions of documents and many customers?
Use the five steps: fix labels and metric, hash features into a cheap linear model, cascade uncertain documents upward, and keep interactive and bulk lanes apart. Store results with the model version, and isolate each customer's data and review queue.
How do you design a Jira bug-to-team classification system?
Treat it as multi-class text classification where the label is the owning team and the input is the title and description. Add an "unsure" route to a triage person, since ownership is ambiguous and changes; a reassignment by an engineer is a free label for retraining.
How would you design RFP analysis with generative AI?
Split the document into requirements, classify each one by type, owner and risk, and use a language model to extract and summarise, with a reviewer checking against the source text. A strong answer measures extraction recall, because a missed requirement is the costly error.
BERT or a smaller model: how do latency, cost and quality trade off?
A distilled model such as DistilBERT keeps most of the quality at lower size and latency, per Sanh et al. (2019). Use the larger model only where the cheaper one's errors are expensive, and measure on your own documents. A prompted LLM needs few labels but costs the most per document and is the slowest, so it suits the last, small slice rather than the first stage.
Which metric do you use when one class is rare?
Use precision, recall, F1 and a precision-recall curve for that class, and pick the threshold from the cost of each error. Accuracy hides the failure, as the glossary example shows, and a search ranking system faces the same choice of metric for rare relevant results.
What changes if latency matters more than quality?
Keep the linear model as the main path, shrink the review lane, cache repeated documents, and accept more abstentions rather than a slow second stage. Say what accuracy you give up.
- A text-classification system at scale is labels, a metric, a cheap first model, a confidence gate and a feedback loop.
- On rare classes use precision, recall, F1 and precision-recall curves, never accuracy alone.
- Feature hashing gives a stateless, fixed-memory first stage at the price of collisions.
- A cascade with abstention spends compute only where the cheap model is unsure.
- Reviewed documents are the best retraining data; monitor per-class precision and the abstention rate.
Sources
- Classification: Accuracy, recall, precision, and related metrics (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
- Machine Learning Glossary: Metrics — Google for Developers, 26 Oct 2026
- Classification: ROC and AUC (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (arXiv 1910.01108) — arXiv (Sanh, Debut, Chaumond, Wolf), 2 Oct 2019
- Bag of Tricks for Efficient Text Classification (arXiv 1607.01759) — arXiv (Joulin, Grave, Bojanowski, Mikolov), 6 Jul 2016
- Feature extraction, scikit-learn 1.9.1 documentation — scikit-learn, 26 Oct 2026
- HashingVectorizer, scikit-learn 1.9.1 documentation — scikit-learn, 26 Oct 2026
