LLM Evaluation

How do you do red-teaming on LLM features?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How do you do red-teaming on large language model (LLM) features? You attack the whole feature, not only the model, as a real attacker or unlucky user would: scope the harms, attack, measure, then fix and re-test. Red-teaming is a structured attempt to make a system fail so the team finds the failure before users do. The next question is what you attack first, and the answer is a written threat model, not a list of jailbreak prompts.

In this article

Before you start
Why interviewers ask this
Red-teaming an LLM feature from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one step of the LLM evaluation learning path.

Why interviewers ask this

Red-teaming questions separate candidates who test a model from candidates who test a product. The question is reported by candidates in loops for LLM feature and evaluation roles. Microsoft's team reports that, as of October 2024, it had run over 80 operations covering more than 100 products (Bullwinkel et al., 2025), so the practice is routine in industry.

A weak answer lists jailbreak prompts. A strong answer starts from harms, treats the feature as a system with retrieval, tools and filters, and says how success is measured. A Staff-level answer adds the limits. Red-teaming finds risk; it does not prove safety. Researchers also report no consensus on its scope, structure or assessment criteria (Feffer et al., 2024), so an interviewer will credit a candidate who states their own definitions.

Red-teaming an LLM feature from first principles

Red-teaming: Red-teaming is a structured, adversarial test in which people emulate real attackers or ordinary users against a whole system to find failures before release, then record each failure with its cause and impact so the team that builds the product can fix it and test again.
Attack success rate: The attack success rate (ASR) is the number of attack attempts that produce the harmful outcome divided by the number of attempts, a fraction between zero and one, measured on one fixed set of attacks so that results before and after a fix can be compared directly.
Break-fix cycle: A break-fix cycle is a repeated loop of red-teaming and mitigation in which each round attacks the system, the product team fixes what broke, and the same attacks run again until the system resists a wide range of attacks. Because a fix can create a new weakness, the loop repeats rather than happening once.

In plain words

Think of a shop owner who pays a locksmith to break in before opening day. The locksmith tries the windows and the back alley, not only the front lock. The locksmith also notices how a confused customer could get locked in. Both matter: a determined burglar and an ordinary visitor who wanders into the wrong room can each cause damage, so the test covers bad intent and honest accidents. The analogy breaks in one place. A lock either opens or it does not. A language model may refuse an attack nine times and comply on the tenth, so you must count outcomes.

Precisely, red-teaming an LLM feature means running realistic attacks against the end-to-end system, counting successes, and retesting fixes.

Step 1: Scope the harms and write a threat model

Red-teaming starts from the harm you fear, not from an attack technique. Bullwinkel et al. (2025) report that starting from downstream impact makes findings more likely to tie to real risk. Ask what the feature can do and where it is used.

A threat model is a short written list of who attacks, how, through which weakness and with what impact. The parts are the system, the actor (a scammer or a typical user), the technique, the weakness and the impact. Impacts are security harms, such as data leaks (see privacy in machine learning (ML) systems), or safety harms, such as bias and unfair outputs.

Example: an assistant that summarizes a user's email. A scammer sends a cross-prompt injection attack (XPIA), which is an email with buried text telling the assistant to ignore its instructions and output a malicious link. The weakness is that the model cannot reliably separate instructions from data; the impact is a malicious link or leaked data. The Open Worldwide Application Security Project (OWASP) Top 10 for LLM Applications is a checklist of categories to cover.

Step 2: Attack the whole system, by hand and with automated attack sets

Attacking the system means testing every input channel and component, not only the model's text box. Bullwinkel et al. (2025) report a vision-language model that refused a direct text request but often complied when the same instruction was overlaid on an image, so each modality needs its own tests. They also report that simple handcrafted attacks often work as well as gradient-based ones, which need model access commercial systems rarely give.

Humans find new categories of harm and judge specialist domains. Automated attack sets add coverage; the paper describes PyRIT (the Python Risk Identification Toolkit), Microsoft's open-source framework with prompt datasets, prompt converters and automated attack strategies. An LLM judge can score outputs, but the paper calls judges less reliable in specialist fields such as medicine, so experts check those.

For scale, Anthropic released 38,961 human red-team attacks and found that models trained with reinforcement learning from human feedback (RLHF) became harder to red-team as they grew, while other model types showed a flat trend (Ganguli et al., 2022).

Step 3: Repeat each attack to estimate how often it works

Repeating each attack many times estimates how often it works, because a language model can fail on one run and pass on the next. Bullwinkel et al. (2025) say testing at scale helps account for non-determinism, and their image case study sent each prompt 50 times.

A worked example, with illustrative counts: 20 attack variants against the email assistant, each run 50 times, is 1,000 model calls.¹ One variant succeeds 7 times in 50 (illustrative), so its attack success rate is 0.14, with a Wilson 95% interval of about 0.07 to 0.26.¹ A Wilson interval is a range, computed from the counts, in which the true rate plausibly lies.

Rare failures hide. At a true failure rate of 0.02, 50 runs find at least one hit with probability 0.636. It takes 149 runs to reach 0.95.¹ Zero hits in 50 runs still leaves an upper bound of about 0.07, so "no hit" is not "safe".¹

A rare failure needs many repeated runs before it is likely to show up even once.CC BY 4.0 — source: How do you do red-teaming on LLM features?

Text description of the diagram

At a true failure rate of 0.02, 50 runs find at least one hit with probability 0.636.
It takes 149 runs to reach 0.95.

Step 4: Fix, retest on the same attacks and keep them as a regression set

After testing, the product team fixes what broke and the red team reruns the same attacks. In Microsoft's setup the red team only identifies risk, while product teams are resourced to build the mitigations (Bullwinkel et al., 2025). Each fix is retested on the same attack set, and the attacks that once worked become a permanent regression set; see golden sets and regression testing.

Counts decide whether a fix worked. If the variant drops from 7 hits to 1 in 50 (illustrative), the rate falls from 0.14 to 0.02, but the intervals, 0.07 to 0.26 and about 0.004 to 0.105, overlap, so 50 runs are too few to show the improvement clearly. With 200 runs each (28 and 4 hits, illustrative), the intervals are 0.10 to 0.195 and 0.008 to 0.050, and they separate.¹

Mitigations can add new risk, which is why the cycle repeats. Plan controls outside the model too: limited permissions and guardrails.

Red-teaming is a loop: each fix is retested on the same attacks that found the problem.CC BY 4.0 — source: How do you do red-teaming on LLM features?

Step-by-step description

Start from the harm you fear and who could cause it.
Try attacks by hand and with automated sets.
Run each attack many times to get a success rate.
Fix, then rerun the same attacks.

Red-teaming starts from the harm you fear, not from an attack technique.
Each fix is retested on the same attack set.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.