In this article
Before you start
This page belongs to the LLM evaluation learning path.
Why interviewers ask this
Anyone can say a model "hallucinates." The consequence of not measuring it is that nobody can tell whether a prompt change, a new model or a retrieval fix made answers more truthful or only more confident. Interviewers ask the question to see whether a candidate can turn a vague worry into a number someone could track.
A weak answer says "a human checks if the answer is right" or "ask another model if it is correct." A strong answer names the unit of judgement, the source of truth and the metric. A Staff-level answer adds what the metric cannot see, how the automatic checker is validated against people, and why the score needs a sample size.
Measuring factuality from first principles
- Atomic fact
- An atomic fact is a short statement that carries exactly one piece of information which can be true or false on its own, such as "she was born in 1950", so a judge can mark it supported or unsupported without splitting it further.
- Factual precision
- Factual precision is the share of the facts a model states that a trusted source supports, so it penalises false statements but not missing ones: an answer that leaves facts out loses nothing.
In plain words
Think of marking an essay. One tick or one cross is useless when the essay has ten claims and seven are right. A fair marker underlines each claim and checks it in a textbook. Then the marker reports seven out of ten. The analogy breaks in one place: a marker also notices what the essay forgot to say, and this score does not.
In one precise sentence, FActScore is a metric that breaks a generation into atomic facts and computes the percentage of them supported by a reliable knowledge source (Min et al., 2023).
The authors say a yes or no verdict is inadequate for long text. One generation mixes supported and unsupported pieces of information. Human evaluation is also slow and costly (Min et al., 2023).
Step 1: Break the answer into atomic facts
The unit of judgement decides what you can learn. A whole answer is too coarse, and so is a sentence: Min et al. (2023, Section 3.1) report that even a single sentence mixes supported and unsupported facts, for example in 40% of cases with ChatGPT.
So you rewrite the answer as a list of atomic facts. In this illustrative sentence, "Ada Lovelace, an English mathematician born in 1815, wrote the first program" becomes three facts: she was an English mathematician, she was born in 1815, and she wrote the first program. A checker may mark the third one not supported, because "first program" is historically debated. A person or a model can do the splitting. Input: one generated answer. Output: a numbered list of single claims.
Take a toy answer of four sentences. Together they hold 10 atomic facts (illustrative).1 You now have countable claims, and the next step gives each one a verdict.
Step 2: Check each fact against a trusted source
Each atomic fact gets one of two labels: supported or not supported by the knowledge source. The source depends on the product. For a biography benchmark it can be an encyclopedia. For a company assistant it is the retrieved documents. For general chat it can be a search engine.
Humans give the most trusted labels but are costly. Min et al. built an automatic estimator that uses retrieval plus a strong language model and report less than a 2% error rate against human scores (abstract). They used it to score 6,500 outputs from 13 models, work that would have cost tens of thousands of dollars if people had done it. That result depends on the retrieval source and the judge model, so validate any automatic checker on a human-labelled sample before you trust it; the page on LLM-as-a-judge covers that check.
Input: the atomic facts and a source. Output: one label per fact.
Step 3: Divide supported facts by all facts
The score is supported atomic facts divided by all atomic facts. In the toy answer, 7 of the 10 facts are supported, so the score is 0.70 (illustrative).1
Now compare grading by sentence. In this toy answer the four sentences hold 3, 2, 3 and 2 facts, of which 2, 2, 3 and 0 are supported. Only two sentences are fully right, so a sentence-level verdict gives 0.50 against 0.70 at fact level.1 The sentence verdict also hides that the first sentence was two-thirds right. The fact-level list shows exactly which claims failed, which is what you fix.
Text description of the diagram
- Judged by whole sentences, 2 of 4 sentences are fully right, a score of 0.50.
- Judged by atomic facts, 7 of 10 are supported, a score of 0.70.
Step 4: Pair the score with a coverage check
Factual precision counts only what the model said. By its definition a model that states two facts, both supported, scores 1.00, while our ten-fact answer scores 0.70.1 The cautious model looks better yet tells you far less.
So report a second number beside it: how much of what a good answer should contain is present, or how often the model declines to answer. Without it, tuning to raise the score teaches the model to say less.
Published work already does this. Min et al. (2023) report the share of prompts each model responds to next to FActScore. The long-form factuality work of Wei et al. (2024) scores answers with F1@K, which combines precision with recall against K expected facts. SimpleQA (OpenAI, 2024) grades short answers as correct, incorrect or not attempted, so abstention is measured directly.
Also report uncertainty. A score of 7 of 10 has a 95% Wilson interval of roughly 0.40 to 0.89, while 70 of 100 narrows it to roughly 0.60 to 0.78 (illustrative).1 Ten facts cannot separate two models; hundreds can.
Step-by-step description
- Split the answer into single claims.
- Mark each claim supported or not.
- Divide supported by all facts.
- Catch answers that say little.
- The answer is split into atomic facts.
- Each atomic fact is checked against a trusted source.
- The score is supported facts divided by all facts.
- A coverage check catches answers that say very little.
The answer is split into atomic facts. Each atomic fact is checked against a trusted source. The score is supported facts divided by all facts. A coverage check catches answers that say very little.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you measure factuality and hallucinations in a language model?
I measure factuality as a fraction, not a verdict. I split each answer into atomic facts, check each fact against a trusted source such as retrieved documents, and report the share supported, which is the FActScore idea. I use a model-based checker for scale, but I compare it with human labels on a sample first. That score is precision only, so I also track coverage or abstention, otherwise a model that says little looks perfect. For retrieval systems I score faithfulness to the retrieved text separately, and I report counts with an interval, because ten facts is a small sample.
What makes it Staff-level: it names the unit, the source, the validation of the automatic judge, the blind spot of the metric and the sampling error.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Unit of judgement | Whole answer marked right or wrong | Atomic facts, since sentences mix true and false claims | Atomic facts, with the splitting step itself spot-checked |
| Source of truth | "A person checks" | A named trusted source per use case | Source chosen per use case, with its own gaps stated |
| Metric | A vague accuracy | Supported facts divided by all facts | Precision plus a recall-style measure such as F1@K, or an abstention rate |
| Trust in the checker | Assumes an automatic judge is right | Compares it with human labels | Reports agreement, sample size and an interval |
| Follow-up readiness | Has no answer to what the metric misses | Names the blind spot: omissions go unpenalised | Offers fixes: F1@K, abstention rate, a golden set |
Three variants of this question
How do you measure hallucinations in a retrieval-augmented system?
Score faithfulness: the share of the answer's claims that the retrieved passages support, whether or not the claim is true in the world; the Ragas faithfulness documentation defines its metric this way. The Ragas page shows how a retrieval-augmented generation (RAG) system is evaluated this way.
How do you detect hallucinations without a reference answer?
Sample the model several times and measure how much the answers disagree; claims that change from sample to sample are more likely to be invented. Semantic entropy is one such method: Farquhar et al. (2024) group sampled answers by meaning and measure the entropy across those groups.
How do you know your automatic factuality judge can be trusted?
Label a sample of facts by hand, run the judge on the same facts, and report how often they agree, with the sample size. Min et al. (2023) did this for their estimator and report less than a 2% error rate.
Common mistakes
- "I grade each answer as correct or incorrect." (wrong) A long answer mixes supported and unsupported facts. (right) "I score the share of atomic facts that a source supports."
- "I judge sentence by sentence." (wrong) A sentence can hold true and false facts together. (right) "I split to atomic facts first."
- "The score is 1.00, so the model is factual." (wrong) The score covers only what was stated. (right) "I also check coverage and how often it abstains."
- "The automatic judge says 0.70, so it is 0.70." (wrong) The judge has its own errors. (right) "I validate it against human labels and report an interval."
Follow-up questions
What is the difference between a hallucination and a factual error?
A factual error is any false statement. Ji et al. (2023) split hallucination into intrinsic, which contradicts the source, and extrinsic, which the source cannot confirm. A hallucination is a statement the model presents as reliable without support from its given source or from fact, so faithfulness to a supplied document and factuality about the world are two different checks. The deeper causes are on the page about why LLMs hallucinate.
Why not ask the model whether its own answer is correct?
The model may share the same blind spots that produced the error, so its self-grade needs the same validation against human labels as any other judge. A sampling-based alternative is SelfCheckGPT (Manakul et al., 2023), which checks whether the model's own samples agree.
How do you reduce hallucinations once you can measure them?
Change one thing at a time, such as better retrieval or a prompt that allows "I don't know", and re-score the same set of questions; the causes and fixes are covered in why LLMs hallucinate.
How do you stop factuality from regressing?
Keep a fixed set of questions with known-good facts and run the score on every change; golden sets and regression testing explains the setup.
Should the metric reward a model for abstaining?
Not through precision alone, which ignores what was left out. Track the abstention rate and coverage next to the score and decide which trade-off the product needs.
- Factuality of long text is a fraction of atomic facts supported, not a yes or no verdict.
- Sentences are too coarse a unit because one sentence mixes true and false facts.
- Validate any automatic checker against human labels before trusting its number.
- Factual precision ignores omissions, so report coverage or abstention beside it.
- Report counts and an interval, because a handful of facts cannot rank models.
Sources
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (arXiv 2305.14251) — arXiv (Min et al., EMNLP 2023), 11 Oct 2023
- FActScore, full text, Section 3 (arXiv 2305.14251 full text) — arXiv (Min et al.), 11 Oct 2023
- Survey of Hallucination in Natural Language Generation (arXiv 2202.03629) — ACM Computing Surveys (Ji et al.), 3 Mar 2023
- Detecting hallucinations in large language models using semantic entropy — Nature 630 (Farquhar et al.), 19 Jun 2024
- Long-form factuality in large language models (arXiv 2403.18802) — arXiv (Wei et al.), 27 Mar 2024
- Measuring short-form factuality in large language models, SimpleQA (arXiv 2411.04368) — arXiv (OpenAI), 7 Nov 2024
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection (arXiv 2303.08896) — arXiv (Manakul et al.), 15 Mar 2023
- Ragas documentation: faithfulness metric (read 2026-10-26) — Ragas documentation, 26 Oct 2026
