Design harmful-content detection

In this article

1. Before you start
2. Why interviewers ask this
3. Harmful-content detection from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one worked problem in the machine learning (ML) system design guide.

Why interviewers ask this

A harmful-content design question tests whether you treat moderation as a system with a policy, a measurement and a human process, not as a single classifier. Both errors hurt: missing hate speech harms people, and wrongly removing a post harms speech. Interviewers listen for how you separate the two and who decides the line.

A weak answer trains one model on a labelled dataset and quotes accuracy. A strong answer names a metric you can actually measure, explains where the labels come from, says what is automated and what goes to people, and describes how the system keeps up when the policy or the attackers change. The prompt takes several forms in system design rounds, such as detecting ads that sell weapons, a fake-news detector or a copyright-violation detector. The skeleton is the same for each.

Harmful-content detection from first principles

Prevalence

Prevalence is the share of views of content on a platform that fall on posts violating a given policy, estimated from a labelled random sample of what people actually saw, not counted from every post.

Multimodal classifier

A multimodal classifier is a model that takes more than one kind of input, such as the words and the pixels of a post, and outputs one score from the combination of the two.

Weak supervision

Weak supervision is a way of creating training labels cheaply by combining many noisy rules or models that vote, instead of having a person label every example.

In plain words

Picture a very large library where a few books are harmful. A guard at the door checks every book, but only the book's cover and a skim of the words. Obvious cases are turned away. Doubtful ones go to a trained librarian. Everything else goes on the shelf. To learn whether the library is safe, someone also picks random books from the shelves and has them read in full. The analogy breaks here: in a feed, harm depends on the pairing of words and picture, and the rules change as people change what they post.

In one precise sentence: a harmful-content system is a policy, a sampled measurement, a label pipeline, a model on every input type, and confidence tiers that send each item to automatic action, human review or nothing.

Step 1: Define harm per policy and set the precision target

The input is the policy text; the output is one classifier per policy, each with a precision target. Harm is not one class. Hate speech, weapon sales and copyright violations differ in what counts, in context and in what action follows, so each gets its own definition, its own labels and its own thresholds. Treat the policy as a versioned document the labelling guide and the model both follow.

The precision target depends on the action. Removing a post automatically needs very high precision, because every false removal is a person silenced. Sending a post to a reviewer tolerates low precision, because a person makes the final call. So a single policy has at least two thresholds. Weapon-sale ads add a second constraint: the rule is about the sale, not the word "gun", so context decides. What you now know: each policy has its own model and its own error costs.

Step 2: Measure prevalence by sampling

Meta defines prevalence as the percentage of times people see violating content, calculated from a sample of content seen and sent to reviewers across languages and regions. It put hate speech at 0.10% to 0.11% for July to September 2020 (Meta, November 2020).

Counting what the classifier flagged measures the classifier, not the harm: it misses what it scores low and counts its own false alarms. A rate is pinned to within a fifth of its value at 95% confidence only once the sample holds about 96 harmful items (1.96 squared divided by 0.2 squared). So a rate of 1% needs about 9,500 labelled items, 0.1% about 96,000 and 0.02% about 480,000.¹

Weight by views: in a toy feed where harmful posts are caught sooner and so seen less, 0.493% of posts are harmful but only 0.127% of views land on them.¹ Count by views or the metric misstates what users meet: here it overstates, and a viral harmful post would make it understate. What you now know: you can only know the harm rate by labelling a sample.

Text description of the diagram

• Measuring a rate of 1% to within 20% takes about 9,508 labelled items.
• Measuring a rate of 0.02% takes about 480,104.

Step 3: Collect labels from reports, reviewers and weak supervision

Labels come from four places: user reports, trained reviewers, the sampled audit of Step 2 and weak supervision. User reports are cheap but biased toward content people notice and are willing to flag. Reviewers are accurate but slow and costly. Weak supervision fills the gap: Snorkel lets experts write labelling functions that express heuristics with unknown accuracies, and denoises their outputs without access to ground truth (Ratner et al., 2017).

The gain has a limit. In a toy run, three independent rules that are each right 70% of the time and vote on half the posts label 71.8% of posts by majority, with 75.8% accuracy on those.¹ The accuracy only holds if the rules err independently, and real rules overlap. Active learning helps: send reviewers the items the model is least sure of, because those teach it most. What you now know: labels are a pipeline with known biases.

Step 4: Build a classifier that reads text and image together

A hateful meme shows why one modality is not enough. The Hateful Memes dataset was built so that only multimodal models can succeed, using benign confounders to block reliance on one signal. A benign confounder is a harmless meme made from a hateful one by swapping only its picture or only its caption, so the part that stayed the same cannot tell the two apart. Then, state-of-the-art methods scored 64.73% accuracy against 84.7% for humans (Kiela et al., 2020).

Early fusion combines text and image features inside one model. Late fusion runs a text model and an image model separately and combines their scores, which is cheaper but misses meaning that exists only in the pair. A practical design runs a cheap model that reads both words and picture on every post, and a larger joint model on its high scores, on reported posts and on posts gaining views fast. A gate built only from single-modality scores would never pass a meme whose parts are each harmless. What you now know: pairs carry meaning that parts do not.

Step 5: Route by confidence and close the loop

The input is the score from Step 4; the output is an action. Above a high threshold the system removes or blocks automatically. Between a low and a high threshold the item goes to a reviewer queue, ordered by score and by expected views. Below the low threshold nothing happens. Reviewer decisions, appeals and the Step 2 audit flow back as labels, which closes the loop to Step 3.

The low threshold sets the review bill. In the illustrative toy run of 1 million posts with 4,934 harmful ones, a review threshold of 0.3 sends 19,344 posts to review and catches 83.5% of harmful posts, while 0.05 sends 182,926 and catches 97.6%.¹ Meta reports that it proactively detects about 95% of the hate speech it removes, meaning before anyone reports it (Meta, November 2020). What you now know: the review threshold is a budget decision.

1 · Define policy2 · Sample prevalence3 · Collect labels4 · Fuse text and image5 · Route by confidenceReplay

Step-by-step description

1. One classifier and target per harm.
2. Label a random sample of views.
3. Reports, reviewers, weak rules.
4. Both inputs read together.
5. Auto, review or leave.

• Each policy has its own classifier and precision target.
• Prevalence is estimated from a labelled sample of views.
• Labels come from reports, reviewers and weak supervision.
• Routing sends high confidence to automatic action and the middle to reviewers.

Worked example

Three calculations, all on illustrative inputs, with the script saved beside the page.¹

Measuring the rate. The sample size for a rate $p$ at a 20% relative half-width and 95% confidence is about $\frac{{1.96}^2(1-p)}{p\times {0.2}^2}$, which gives 9,508, 95,944, 191,984 and 480,104 items for rates of 1%, 0.1%, 0.05% and 0.02%.¹

The review budget. An illustrative toy classifier scores 1 million posts, 4,934 of them harmful. Automatic removal above 0.98 takes down 345 posts, all harmful in this run. The table varies the review threshold below it, assuming 30 seconds of reviewer time for each item.

Here "caught" means removed automatically or sent to a reviewer, so it assumes reviewers decide correctly. The chart below spaces the four hour values equally; it is not to scale.

Text description of the diagram

• A review threshold of 0.3 catches 83.5% of harmful posts for 161 reviewer hours.
• A threshold of 0.02 catches 99.3% for 3,024 hours.

The weak labels are in Step 3: three rules that are each 70% accurate give 75.8% majority accuracy on 71.8% of posts, if they err independently.¹

Trade-offs and edge cases

Every moderation design trades missed harm against wrongly removed speech, and the trade is made per policy and per action.

• Precision against recall. Automatic removal wants precision near 100%; the review band buys recall at a labour cost, as the table shows.
• Policy drift and new harms. A new kind of harm, such as a new scam format, has no labels yet. Weak rules and the sampled audit find it first; then retraining catches up. See distribution shift.
• Adversaries. People alter text and images to slip past a model, so use embedding-based similarity to known violations as a second signal (embeddings and vector search) and retrain often.
• Languages and context. Reviewers and models must cover each language; Meta sends samples to reviewers across languages and regions because hate speech depends on language and cultural context (Meta, November 2020).
• The 2025–26 state: policy-taxonomy safeguards. Newer safeguards are multimodal language models that read an image and its text against a written harm taxonomy. Llama Guard 3 Vision (Meta, November 2024) is one: it classifies the text-and-image prompts and the text replies of a human–AI image conversation, and its authors tested it on the MLCommons taxonomy (Chi et al., 2024). It is a safeguard for assistant conversations, not a feed classifier, so treat it as a second-stage model that you would still need to measure with the Step 2 sample.
• Appeals. A successful appeal is a corrected label that enters training and shows where a threshold is too strict; appeals are a biased sample, since only removed posts and only people who choose to appeal are in it.
• Ranking, not only removal. Some borderline content is better demoted in the feed than removed, which makes the action a score, not a switch.

For the surrounding serving questions see batch, streaming and real-time inference, the feature store walkthrough and machine learning operations (MLOps).

How to answer it in an interview

The spoken structure: policy, metric, labels, model, tiers, loop.

Common mistakes

• "I would train one classifier for harmful content." (wrong) Policies differ in definition, context and action. (right) "I would build one classifier per policy with its own thresholds."
• "I would report accuracy." (wrong) Harm is rare, so a model that flags nothing scores high. (right) "I would report prevalence from a sample, plus precision and recall per tier."
• "I would automate every decision." (wrong) False removals silence people and there is no human check. (right) "I would auto-act only at very high confidence and route the middle to reviewers."
• "I would fuse text and image after scoring each." (wrong) Some harm exists only in the pairing. (right) "I would make the first-stage model read text and image together, and send its high scores to a larger joint model."

Follow-up questions

Design an end-to-end ML solution to detect ads selling weapons?

Define the policy as the sale of a weapon, not the word, then add a first-stage model that reads the ad's text and image together, with a larger joint model on high scores and reported ads. Use weak rules for labels, review the middle band and measure prevalence by sampling ad views.

Design a weapon ad detection system?

Start from the policy and its precision targets, then treat the ad's text, image and landing page as three inputs read together. Remove automatically only at very high confidence and send the rest to reviewers.

Design an image copyright-violation detection system?

The key difference is that a match is against a known reference, so similarity search over embeddings or fingerprints does most of the work, with reviewers for edits and fair-use cases. Precision matters because a takedown is costly to reverse.

Design a fake news detection system?

Truth is not visible in the post alone, so combine content signals, source history and spread patterns with fact-checker labels, and prefer demotion with a label over removal. Expect slow labels, because fact-checkers take time.

What if the harm rate is 0.01%?

The sample must be ten times larger: about 960,000 labelled items instead of 96,000, because you still need about 96 harmful items in it.¹

Should text and image be fused early or late?

Early fusion learns interactions and catches pair-only meaning; late fusion is cheaper and modular but can miss it. Use a cheap early-fusion model for coverage on every post and a larger joint model on its high scores.

Sources

1. Measuring Progress in Combating Hate Speech (Meta; page updated December 2021) — Meta, 19 Nov 2020
2. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (arXiv 2005.04790v3) — arXiv (Kiela et al.), 7 Apr 2021
3. Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations (arXiv 2411.10414v1) — arXiv (Chi et al.), 15 Nov 2024
4. Snorkel: Rapid Training Data Creation with Weak Supervision (arXiv 1711.10160) — arXiv (Ratner et al.), 28 Nov 2017