Why do LLMs hallucinate, and how do you reduce it?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM hallucination from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Hallucination is one of the failure modes covered in the LLM evaluation guide.

Why interviewers ask this

A candidate who answers "lower the temperature" shows they have used a model but not studied its failures. Candidates report being told that exact answer was wrong when asked how to stop a model from hallucinating, and the question is reported in interviews at AI labs and large tech companies, often as "what causes it and how do you reduce it in a production feature?".

A weak answer calls hallucination a bug or a data-quality problem. A strong answer separates the cause (training and grading) from the symptom (a confident error) and names mitigations that act on each. A Staff-level answer adds how you would measure the error rate, because a fix you cannot measure is a hope.

LLM hallucination from first principles

Hallucination

A hallucination is a statement from a language model that sounds plausible and is stated with confidence but is incorrect. The word is borrowed from psychology; the model perceives nothing. What separates it from an honest "I don't know" is the confidence, and what separates it from a typo is that the error looks believable.

Negative marking

Negative marking is a grading rule that subtracts points for a wrong answer and gives zero for a blank, so a guess is worth making only when the chance of being right is high enough. Some standardized tests have long used it to discourage blind guessing.

In plain words

Picture a student in an exam where a blank scores zero and a wrong answer also scores zero. The student writes something on every question. Over many questions, the guesser beats the honest student.

A language model is in that exam. The analogy breaks in one place: the student knows when they are guessing, while a model has no built-in flag for it unless training adds one.

In one precise sentence, a language model hallucinates when it produces the most plausible continuation for a question whose answer it cannot reliably know, and nothing in training or grading told it to stop.

Step 1: Pretraining leaves errors on arbitrary facts

Pretraining teaches a model to output the next word of huge amounts of text. Kalai et al. (2025) argue that "Hallucinations need not be mysterious—they originate simply as errors in binary classification," where the classification is valid statement versus error.

Input: a question about a fact. Output: a fluent answer. Facts that follow a pattern, such as spelling or matching brackets, are learned and their errors shrink as models grow. OpenAI's post on the paper gives the contrast: labeling pet photos as cat or dog is learnable, but labeling them by birthday is not, because birthdays are essentially random. As illustrative reasoning, a birthday that appears once in the training text is the typical case: the model has seen it once and has nothing to check it against. Arbitrary, low-frequency facts have no pattern to learn, so they are where invented answers appear. The model also cannot tell such a gap from knowledge, because both produce fluent text.

Step 2: Grading rewards a guess over saying I don't know

Most benchmarks grade only accuracy, the share of answers that are exactly right. The OpenAI post says that when models are graded this way they are encouraged to guess rather than say "I don't know", and that models are optimized to be good test-takers. Kalai et al. put it as training and evaluation procedures that reward guessing over acknowledging uncertainty. Post-training therefore keeps reinforcing confident guesses that pretraining started.

This is why lowering the temperature does not fix it. Temperature is a decoding setting that flattens or sharpens the probabilities from which the next word is drawn. A low temperature makes the model repeat its single most likely answer. If that answer is a wrong birthday, it now repeats the wrong birthday every time. This is reasoning from how decoding works, not a measured result: temperature changes how the model chooses among what it believes, not what it knows or whether it abstains.

Step 3: Fixing it means scoring uncertainty and grounding answers

The fix Kalai et al. propose is to change the scoring of the existing benchmarks that dominate leaderboards, not to add more hallucination tests. The OpenAI post states the rule: penalize confident errors more than uncertainty, and give partial credit for appropriate expressions of uncertainty.

A model can then abstain when its confidence is low. Grounding adds a second layer. Retrieval-augmented generation (RAG) is a method that fetches documents relevant to the question and puts them in the prompt, so the model can copy from text instead of recalling a fact from memory. Grounding helps when the answer is in the documents; the variants below cover when it still fails. You now have the full chain: gaps in pretraining, a grading rule that pays for guessing, and two levers, scoring and grounding.

1 · Pretraining gaps2 · Guess-friendly grading3 · Score uncertaintyReplay

Step-by-step description

1. Arbitrary facts leave errors behind.
2. A guess scores more than a blank.
3. Penalize confident errors, credit abstaining.

• Arbitrary facts with no pattern leave errors after pretraining.
• Accuracy-only grading rewards a guess over saying I don't know.
• The fix is to score confident errors worse than uncertainty.

Worked numbers: what a guess is worth

Take the OpenAI post's birthday case: a guess has a 1-in-365 chance of being right, and "I don't know" scores zero. Under accuracy-only grading the guess is worth 0.0027 points in expectation and abstaining is worth 0, so the guess never loses.¹ Now add negative marking with an illustrative penalty of 3 points for a wrong answer. A guess is then worth 4p − 3 for a chance p of being right, which is −2.989 at 1 in 365, and it breaks even only at p = 0.75.¹

The same effect shows in real models. In the OpenAI post (results on SimpleQA, a test of short fact-seeking questions, taken from the generative pre-trained transformer (GPT) 5 system card), gpt-5-thinking-mini abstains on 52% of questions, is right on 22% and wrong on 26%, while o4-mini abstains on 1%, is right on 24% and wrong on 75%.

Accuracy-only grading ranks o4-mini first; penalized grading reverses it.¹ Of the questions each model attempted, 54.2% of the answers from gpt-5-thinking-mini were wrong against 75.8% from o4-mini.¹

Text description of the diagram

• gpt-5-thinking-mini abstains on 52% of questions and is wrong on 26%.
• o4-mini abstains on 1% and is wrong on 75%.
• Accuracy is 22% against 24%.

The Staff-level answer

What makes it Staff-level: it names both causes, rejects the temperature myth with a reason, and ends with how to measure.

The rubric

Three variants of this question

How do you detect hallucinations rather than prevent them?

Compare each answer with trusted text, or ask the same question several times and flag answers whose meanings disagree, as in semantic entropy. A second model can grade answers against the source, as in LLM-as-a-judge, and Ragas scores how faithful an answer is to its retrieved context.

How do you reduce hallucinations in a production feature?

Ground answers in your own documents, let the model say it does not know below a confidence threshold, show sources, and send high-risk topics to a person. Then measure the wrong-answer rate on a labeled set, not accuracy alone.

Why do LLMs hallucinate even with RAG?

Retrieval gives the model text, but the model still writes the answer word by word. If retrieval misses the right passage, the gap from step 1 remains. If the passage is there, the model can still ignore or distort it, and retrieved text can carry injected instructions (see AI agent security). This is reasoning from the pipeline's design, so test each stage.

Common mistakes

• "Lower the temperature and it stops hallucinating." (wrong) Temperature changes how the model picks among its own beliefs, not what it knows. A low temperature can cut errors caused by sampling an unlikely word when the top answer is right, but it cannot fix a wrong top answer or make the model abstain. (right) "Temperature trades variety for repeatability; abstention and grounding address the cause."
• "A bigger model will not hallucinate." (wrong) The OpenAI post says accuracy never reaches 100% whatever the model size, because some questions cannot be answered. (right) "Scale lowers errors on learnable facts, but arbitrary facts stay."
• "RAG removes hallucinations." (wrong) Retrieval can miss or be ignored. (right) "RAG reduces them when the answer is in the documents; I measure faithfulness separately."
• "Add one hallucination benchmark and we are done." (wrong) The OpenAI post says a good hallucination eval has little effect against many accuracy-based evals that reward guessing. (right) "The main scoring has to credit uncertainty."

Follow-up questions

Are hallucinations inevitable?

Not according to the OpenAI post, because a model can abstain when it is uncertain. What is not possible is a model that is always right, since some questions are unanswerable.

Does a bigger model hallucinate less?

Not necessarily. The OpenAI post notes it can be easier for a small model to know its limits, because being calibrated needs far less computation than being accurate.

How do you make an LLM say "I don't know"?

Train and grade so that abstaining beats a low-confidence guess, for example with negative marking, and set a confidence threshold below which the system declines. Prompting alone gives a weaker version of the same effect.

Is hallucination rate on a benchmark the same as in production?

No. A benchmark question set differs from real traffic, so measure on a labeled sample of live queries too. The LLM evaluation guide covers both.

Sources

1. Why Language Models Hallucinate (arXiv 2509.04664) — arXiv (Kalai, Nachum, Vempala, Zhang), 4 Sep 2025
2. Why language models hallucinate — OpenAI, 5 Sep 2025