How do you make an LLM say I don't know?

In this article

1. Before you start
2. Why interviewers ask this
3. Making a model say I don't know from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the generative AI system design interview guide learning path.

Why interviewers ask this

A question-answering product that cannot say I don't know will sooner or later state a falsehood to a user in confident language. A design round on retrieval or chat products can reach this topic because it separates candidates who treat abstention as a designed behavior from candidates who treat it as a prompt afterthought.

A weak answer says "add 'don't hallucinate' to the prompt." A strong answer explains why models guess, names three outcomes (correct, wrong, abstain), sets a threshold, grounds answers in documents and measures the result. A Staff-level answer adds the cost side: every abstention is a lost answer, so the threshold is a product decision backed by measured rates, not a slogan. The sections below build that answer from zero.

Making a model say I don't know from first principles

Abstention

Abstention is a model's choice to give no answer, in words such as "I don't know", because the expected cost of a wrong guess outweighs the expected gain of a right one; it counts as a third outcome beside a correct answer and a wrong one.

Semantic entropy

Semantic entropy is a measure of how much several sampled answers to one question disagree in meaning rather than in wording, so a high value flags an arbitrary answer that is likely a confabulation.

In plain words

Picture a student in an exam where a blank answer scores zero and a wrong answer also scores zero. The sensible student writes something on every question. Kalai et al. (2025) use the same comparison: language models, like students facing hard exam questions, guess when uncertain. The analogy breaks in one place: a student feels doubt, while a model has no built-in doubt signal, so we must measure or train one.

In one precise sentence: a model says I don't know only when its score rules, its prompt and its context make abstaining the better choice than a guess. So the fix is rarely one magic instruction. It is a design with three parts: how answers are scored, what the model is told and given to read, and how often it is wrong, right or silent when tested.

Step 1: Accuracy-only grading rewards guessing

Accuracy-only grading rewards a guess over an abstention. A hallucination is a plausible but false statement that a language model states as fact, and a model graded only on the share of exactly right answers is encouraged to guess rather than say I don't know (OpenAI, 2025).

Take a question about a birthday the model cannot know. A guess has a 1-in-365 chance of being right, about 0.27%, and saying I don't know scores zero.¹ Across many questions the guesser still wins the scoreboard. The authors of the paper argue that "training and evaluation procedures reward guessing over acknowledging uncertainty" (Kalai et al., 2025). A second cause sits in pretraining: arbitrary low-frequency facts, like a pet's birthday, "cannot be predicted from patterns alone" (OpenAI, 2025).

Input: a question with no recoverable answer. Output: a fluent guess. You now know why the default is to answer; the next step changes the score.

Step 2: A penalty for wrong answers sets the confidence threshold

A penalty p for wrong answers sets the threshold p over one plus p. A confidence threshold is the level of confidence below which a system should abstain instead of answering. The same research post advises that you "penalize confident errors more than you penalize uncertainty" and give partial credit for appropriate expressions of uncertainty (OpenAI, 2025).

The threshold below is the same rule Kalai et al. give as an explicit confidence target. Score +1 for a correct answer, 0 for abstaining and −p for a wrong answer. A guess that is right with probability q has an expected score of q minus p times (1 minus q). That is positive exactly when q is greater than p/(1+p).¹

With p = 0, which is accuracy-only grading, the threshold is zero and every guess pays. With p = 1 the threshold is 0.5, with p = 3 it is 0.75 and with p = 9 it is 0.9.¹ Input: a penalty. Output: the confidence below which to abstain. Raising the penalty is the lever that moves the model from guessing to staying silent.

A penalty reaches a deployed model by three routes. First, state the target in the instruction: Kalai et al. (2025) propose prompts of the form "Answer only if you are > t confident, since mistakes are penalized t/(1−t) points", and t/(1−t) equals the penalty p above. Second, gate at the system level: compute a confidence signal, such as a self-evaluation probability or semantic entropy, and withhold the answer when it falls below the threshold. Third, use the three-outcome score in evaluations and training rewards, which is the main recommendation of the research post.

Step 3: The prompt and the context make abstaining easy to choose

Permission, grounding and quotes make abstaining easy to choose. Grounding is the practice of tying every claim in an answer to text that was supplied to the model. The Anthropic documentation (as of October 2026) lists the levers: explicitly allow the model to say it does not know, restrict it to the provided documents instead of its general knowledge, and for long documents (over 20k tokens) ask for word-for-word quotes first. If it cannot find a quote, it must retract the claim.

Answer only from the documents below.
For each claim, quote the supporting sentence first.
If no quote supports a claim, drop the claim.
If the documents do not contain the answer, reply: I don't know.

Relevant context also helps the model itself: Kadavath et al. (2022) find that a trained "I know" probability rises appropriately when relevant source material is in the context. The documentation adds that these techniques significantly reduce hallucinations but do not eliminate them. For retrieval design, see how to design RAG over real documents.

Step 4: Sampled checks and measured rates verify the behavior

Answer, wrong and abstain rates measure the result. A prompt change is a claim until a test set shows it works, so build questions that include some the documents cannot answer, then count answered, wrong and abstained cases.

Two checks flag shaky answers. The documentation suggests running the same prompt several times and comparing outputs, because inconsistencies "could indicate hallucinations" (Anthropic, as of October 2026). Farquhar et al. (2024) refine this with semantic entropy, computed at the level of meaning rather than word sequences, to detect confabulations: fluent claims that are wrong and arbitrary. Model-reported confidence needs checking. Kadavath et al. (2022) found self-evaluation, called P(True), well calibrated on many tasks. A trained "I know" probability, P(IK), predicts well and partially generalizes, but it struggles with calibration on new tasks. Validate any confidence signal on your own questions before you threshold on it.

Input: a test set and the system's answers. Output: three rates. The stage-by-stage RAG evaluation guide shows where to place them.

1 · Guessing wins2 · Set a threshold3 · Permit and ground4 · MeasureReplay

Step-by-step description

1. Accuracy-only grading rewards guessing.
2. A penalty sets the confidence threshold.
3. Prompt and context make abstaining easy.
4. Three rates verify the behavior.

• Accuracy-only grading rewards a guess over an abstention.
• A penalty p for wrong answers sets the threshold p over one plus p.
• Permission, grounding and quotes make abstaining easy to choose.
• Answer, wrong and abstain rates measure the result.

Worked example

Two small examples show the numbers. First, a reported benchmark: on the SimpleQA table in the OpenAI post, gpt-5-thinking-mini abstained on 52% of questions, was right on 22% and wrong on 26%. The model o4-mini abstained on 1%, was right on 24% and wrong on 75%. The accuracy gap is only 2 points, but the error gap is about 2.9 times (75 against 26).¹ Of the questions each model attempted, 54.17% were wrong for the first and 75.76% for the second.¹

Second, a toy to show the threshold at work. Assume, illustratively, that the model's confidence is calibrated and spread between zero and one with density 6c(1−c) (a Beta(2,2) distribution, symmetric around 0.5), and that a wrong answer costs a penalty of 3, so the best threshold is 0.75.¹ Per 100 questions, illustrative:

The thresholds are computed in the script named in the footnote;¹ the SimpleQA rates are from the OpenAI post.

At threshold 0 the model is wrong on 50 of 100 questions (illustrative). At threshold 0.75 it abstains on 84.4 of 100 questions and is wrong on 2.5 (illustrative).¹ A higher threshold trades wrong answers for abstentions, so the threshold is a product decision, and the toy also shows the cost: the model answers only 15.6 of 100 questions.

Text description of the diagram

• At threshold 0 the model is wrong on 50 of 100 questions (illustrative).
• At threshold 0.75 it abstains on 84.4 of 100 questions and is wrong on 2.5 (illustrative).

Trade-offs and edge cases

No single lever is enough. Permission prompts reduce hallucinations but do not eliminate them, quote-first grounding suits long documents, a document-only rule depends on good retrieval, best-of-N compares repeated answers and flags inconsistency, and semantic entropy covers only confabulations, not every kind of hallucination (Farquhar et al., 2024). Prompts are not the only lever: Kalai et al. (2025) argue that evaluations should credit abstention, and a training reward that penalizes confident errors moves the same trade-off into the model itself.

Text description of the diagram

• Best-of-N compares repeated answers and flags inconsistency.
• Semantic entropy covers only confabulations, not every kind of hallucination.

Edge cases to name in a design round:

• Over-abstaining. The toy shows an answer rate of 15.6% at the highest threshold. A sourced, quantified helpfulness cost is not available, so measure it on your own question set.
• Unanswerable questions. Accuracy will never reach 100% because some questions are inherently unanswerable (OpenAI, 2025); hallucinations are not inevitable, because a model can abstain.
• Model size. It can be easier for a small model to know its limits, and calibration needs less computation than accuracy (same source).
• Retrieval versus model. An empty or wrong retrieval looks like a model failure; test them separately, and see agentic RAG for loops that retry retrieval before giving up.
• Permissions. When the answer exists but the user may not see it, the reply must not leak it; see permission-aware RAG.
• Cost. Repeated sampling multiplies calls; see making RAG fast and cheap.

How to answer it in an interview

The spoken structure: say why models guess, name the three outcomes and the threshold, give the prompt and grounding levers, then the measurement and the cost of abstaining too often.

Common mistakes

• "I added 'don't hallucinate' to the prompt, so it is solved." (wrong) The vendor documentation says such techniques reduce hallucinations but do not eliminate them. (right) "I combine permission, grounding and a check, and validate critical outputs."
• "Higher accuracy means a more trustworthy model." (wrong) In the SimpleQA table, 2 points more accuracy came with about 2.9 times the error rate.¹ (right) "I compare error and abstention rates, not accuracy alone."
• "Any variation between answers is a hallucination." (wrong) Different wording can carry the same meaning. (right) "I compare meaning, as semantic entropy does."
• "The model's stated confidence is reliable." (wrong) A trained "I know" probability struggles with calibration on new tasks. (right) "I validate any confidence signal on my own data before using it as a threshold."

Follow-up questions

Can you stop a language model from hallucinating completely?

No. Hallucinations are not inevitable, since a model can abstain, but accuracy cannot reach 100% because some questions are unanswerable, and the Anthropic documentation says its techniques do not eliminate hallucinations. Aim to lower the wrong-answer rate and validate critical outputs.

Does asking the model how confident it is work?

Partly. Kadavath et al. (2022) found self-evaluation, P(True), well calibrated on many tasks, while a trained "I know" probability predicts well but struggles with calibration on new tasks. Validate any confidence signal on your own questions before you threshold on it.

What should a RAG system do when retrieval finds nothing?

Instruct the model to answer only from the supplied documents and to say I don't know when they lack the answer, then log those cases as retrieval misses. The generative AI system design interview guide places this inside the full design.

How do you stop a model from refusing too often?

Lower the threshold or the penalty, and watch the answer rate on answerable questions. The toy shows a model answering only 15.6 of 100 questions at the strictest setting (illustrative), so tune against your measured rates.

Does a bigger model know its limits better?

Not necessarily. Kadavath et al. (2022) found that calibration improves with model size, yet the OpenAI post finds that it can be easier for a small model to know its limits, so size alone does not settle it.

How do you test that abstention works?

Build a question set with some unanswerable items, then report answered-correct, wrong and abstained rates before and after each change, and compare both the wrong rate and the answer rate.

Sources

1. Why language models hallucinate — OpenAI, 5 Sep 2025
2. Reduce hallucinations (Claude Platform Docs) — Anthropic, 26 Oct 2026
3. Why Language Models Hallucinate (arXiv 2509.04664) — arXiv (Kalai, Nachum, Vempala, Zhang), 4 Sep 2025
4. Language Models (Mostly) Know What They Know (arXiv 2207.05221) — arXiv (Kadavath et al.), 11 Jul 2022
5. Detecting hallucinations in large language models using semantic entropy — Nature 630:625-630 (Farquhar, Kossen, Kuhn, Gal), 19 Jun 2024