BLEU vs ROUGE: what each metric measures and when to use it

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the classical machine learning guide, next to the other tools for judging a model.

Why interviewers ask this

Text-generation metrics are where a candidate shows whether they trust a number or understand it. The question arrives as "explain BLEU, ROUGE and BERTScore and their limits", as "how would you evaluate a summarizer", or as "your translation model improved by two BLEU points, should we ship it?". Each one tests whether you know what is being counted.

A weak answer lists names and says "higher is better". A strong answer says that BLEU is precision with a length penalty, that ROUGE is recall, that both count overlapping words and so miss paraphrases, and that the next step is an embedding-based or learned metric plus a human check. Interviewers in the evaluation and applied machine learning loops use this topic to separate people who have read an evaluation report from people who have had to decide from one.

What each one is

BLEU

BLEU, short for bilingual evaluation understudy, is a metric that scores a machine translation by the share of its n-grams that also appear in one or more human reference translations, multiplied by a penalty for output that is too short.

ROUGE

ROUGE, short for recall-oriented understudy for gisting evaluation, is a family of metrics that score a generated summary by the share of the reference summaries' n-grams, or longest word sequences, that the summary also contains.

N-gram

An n-gram is a run of n consecutive tokens in a text, so a unigram is one word, a bigram is two words in a row, and a four-gram is four words in a row.

BERTScore

BERTScore, built on BERT (a pretrained language model), is an evaluation metric that compares each token of a generated text with the tokens of a reference using contextual embeddings from a language model, instead of requiring identical words.

BLEU in plain words

BLEU asks one question of a translation: of the words and phrases you wrote, how many would a human translator also have written? It counts matching n-grams and divides by the number the candidate produced, which is precision. Papineni et al. (2002) built it as "an automated understudy to skilled human judges", cheap enough to run every day while a system is being developed.

Two fixes keep precision honest. First, counts are clipped: a word can only be credited as many times as it appears in the reference, so a candidate that repeats "the" seven times earns 2 of 7 against a reference containing "the" twice, the paper's own example. Second, a brevity penalty, an exponential factor that shrinks the score when the candidate is shorter than the reference, stops a very short output from winning on precision alone. The final score multiplies that penalty by the geometric mean of the four precisions, for runs of one to four words: multiply the four precisions together and take the fourth root,

Because it is a product, one precision of 0 makes the whole score 0. BLEU never checks whether a sentence is true, only whether its word runs were seen.

ROUGE in plain words

ROUGE asks the opposite question of a summary: of the content in the human reference, how much did the summary keep? Lin (2004) describes ROUGE-N as an n-gram recall, because the denominator counts the reference side, and contrasts it directly with BLEU as a precision-based measure.

The package has several members. ROUGE-N counts overlapping n-grams of one size, so ROUGE-1 counts words and ROUGE-2 counts word pairs. ROUGE-L uses the longest common subsequence, the longest list of words in both texts in the same order but not necessarily side by side, so it rewards keeping the order of ideas. ROUGE-W weights consecutive matches more, and ROUGE-S counts skip-bigrams, word pairs in order with gaps allowed. Reports often add an F score that combines recall with precision.

Recall has the opposite weakness: a longer summary covers more of the reference, so evaluations fix the length (the Document Understanding Conference (DUC) tasks in Lin (2004) asked for fixed lengths, from about 10 to 400 words) or report an F score.

Why a correct paraphrase can score zero

BLEU and ROUGE count identical tokens, so a sentence that means the same thing in different words loses credit. Take the reference "the cat sat on the mat today" and the paraphrase "a cat was sitting on a mat today". A person would accept it. Matching words are "cat", "on", "mat" and "today", so one-word precision is 0.5, two-word precision is 0.14 (only "mat today" matches), and no run of three or four words matches, so those precisions are 0. The product of 0.5, 0.14, 0 and 0 is 0, and its fourth root is 0, so BLEU-4 is exactly 0.0.¹

The Papineni paper notes that the geometric average is harsh if any precision vanishes, and treats that as rare on a large test set. That is why BLEU is meant for a whole corpus of sentences, not one sentence. ROUGE does not collapse in the same way because each ROUGE score is one ratio for one n-gram size or the longest common subsequence, so there is no product for a zero to wipe out, but it still cannot see that "sitting" and "sat" are close in meaning. BERTScore exists for this gap. Zhang et al. (2020) compare tokens through contextual embeddings and report better agreement with human judgements than earlier metrics on 363 translation and captioning systems.

Worked example

One reference, "the cat sat on the mat today", and five candidates, scored with BLEU-4 and ROUGE from their definitions in a short script, not by hand.¹

Read the rows in turn. The exact copy scores 1 everywhere. The one-swap candidate keeps six of seven words in order, so ROUGE-1 and ROUGE-L are 0.857, but one swapped word cuts BLEU-4 to 0.489, because the swap breaks every four-word run that crosses it, with precisions of 0.857, 0.667, 0.4 and 0.25 for one to four words.¹ The paraphrase scores 0 on BLEU-4 yet 0.533 on ROUGE-1 F1, because ROUGE-1 only needs words to match, not runs.

The short prefix shows the brevity penalty at work. Its precisions are 1, 1 and 1 for one to three words, but the candidate has 3 words against a reference of 7, so the penalty is $e^{1-7/3}$, about 0.264. The four-gram precision is 0 of 0; with no smoothing (adding a small count so that one zero precision does not zero the product) it counts as 0, so BLEU-4 is 0. The penalty shows alone in BLEU-2, where every precision is 1: BLEU-2 falls to 0.264. ROUGE-1 recall is 0.429 because only 3 of 7 reference words are covered, which is recall working as intended.¹

The repeat candidate shows clipping: of seven "the", only 2 are credited, so unigram precision is 2 of 7.¹

Text description of the diagram

• One swapped word cuts BLEU-4 to 0.489 and the paraphrase scores 0 on BLEU-4.
• The paraphrase scores 0.533 on ROUGE-1 F1.

Text description of the diagram

• BLEU is a precision-based measure and ROUGE-N is a recall-related measure.
• BLEU was built for machine translation and ROUGE for summaries.

Trade-offs and edge cases

Overlap metrics are cheap and repeatable, which is their value, and they are blind to meaning, which is their limit. Use them to track one system through a series of changes on a fixed test set, not to prove that a paraphrasing system is wrong.

Why does the same system get different BLEU scores in different papers?

BLEU is a parameterized metric, and unreported settings change the score. Post (2018) finds differences of up to 1.8 BLEU between commonly used configurations, caused mainly by how the reference text is split into tokens and normalized, and proposes SacreBLEU so that the settings are fixed and printed. When you compare numbers across papers, check that tokenization and the number of references match.

What does a multi-reference score change?

Both papers allow several references. BLEU credits an n-gram if any reference contains it, which raises scores because a good translation has many valid wordings. ROUGE takes the best score over the references in its multiple-reference form. Scores from one reference and from four references are not comparable. In the original paper a human translator scored 0.3468 against four references and 0.2571 against two on the same test set (Papineni et al., 2002, §3), and Lin (2004, §2.1) describes the multiple-reference form of ROUGE.

Should a two-point BLEU gain decide a release?

No. Kocmi et al. (Nov 2021) examined 4,380 translation systems and warn that "sole use of BLEU" can lead to bad deployment decisions. The 2022 metrics task, Freitag et al. (Dec 2022) found that overlap metrics correlate poorly with human ratings and that learned neural metrics did better. So report BLEU for comparability, but base the decision on a learned metric, one trained on human ratings, plus human review of a sample.

When does each metric fit?

Report BLEU for translation (next to a learned metric), ROUGE for summaries, and neither alone for open-ended generation such as dialogue, where many different replies are right and overlap with one reference says little. For those tasks use an embedding metric such as BERTScore, a model-based judge, and human review of a sample. The flow below summarizes the choice. It uses the same precision and recall idea as the confusion matrix, and the probability and statistics guide shows how to size the human sample.

Text description of the diagram

• Translation reports BLEU next to a learned metric; summaries use ROUGE.
• Open-ended text needs an embedding metric, a judge model and human review.

How to answer it in an interview

The spoken structure is: precision against recall, the length fix, the blind spot, then what you add.

Common mistakes

• "BLEU and ROUGE are basically the same metric" (wrong). They count overlap from opposite sides, precision against recall (right: BLEU asks how much of the output is correct, ROUGE asks how much of the reference is covered).
• "A higher BLEU means a better translation" (wrong). The score depends on tokenization, references and the test set (right: compare only under identical settings, for example with SacreBLEU).
• "ROUGE measures how good a summary is" (wrong). It measures overlap with a reference, so a fluent, correct summary in new words scores low (right: pair it with human review and an embedding metric).
• "BLEU works well on one sentence" (wrong). If not one four-word run matches, sentence-level BLEU-4 is exactly 0 without smoothing (right: report it over a corpus, or use a smoothed sentence score).
• "BERTScore replaces human evaluation" (wrong). It correlates better with people on the tested systems but is still a proxy (right: use it as one signal among several).

Follow-up questions

Explain BLEU, ROUGE and BERTScore. When would you use each, and what are their limits?

BLEU is clipped n-gram precision with a brevity penalty, used for translation; ROUGE is n-gram or subsequence recall, used for summaries; BERTScore matches tokens by embedding similarity, so it credits paraphrases. The shared limit is that all three score a reference, so a valid answer unlike the reference loses credit.

What is the BLEU score and its significance?

BLEU is a number from 0 to 1, often shown from 0 to 100, that rises as a translation shares more word runs with human references. Its significance is historical and practical: it gave translation research a quick, repeatable substitute for slow human judging, and it is still reported next to newer metrics.

What happens to a BLEU score with more references?

It rises. In the original paper a human translator scored 0.3468 against four references and 0.2571 against two on the same test set (Papineni et al., 2002, §3), so scores from different numbers of references are not comparable.

Can you use ROUGE to evaluate a chatbot?

Poorly. A chatbot reply can be correct in many wordings, so overlap with a single reference undercounts good answers. Use a judge model, task-success checks and human review, and keep ROUGE for summary-like outputs. The transformers guide covers the models that generate such text.

How would you evaluate a summarizer when there are too many outputs to read?

Sample outputs, have people score a small stratified set for faithfulness and coverage, and use ROUGE and an embedding metric only to monitor change between versions. Make the sample large enough for a confidence interval; the probability and statistics guide shows how.

Sources

1. Bleu: a Method for Automatic Evaluation of Machine Translation — ACL 2002 (Papineni, Roukos, Ward, Zhu); year and month only, 1 Jul 2002
2. ROUGE: A Package for Automatic Evaluation of Summaries — ACL Text Summarization Branches Out (Lin); year and month only, 1 Jul 2004
3. BERTScore: Evaluating Text Generation with BERT — arXiv (Zhang et al.), v3, 24 Feb 2020
4. A Call for Clarity in Reporting BLEU Scores — arXiv (Post), v2, 12 Sep 2018
5. To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation — WMT 2021 (Kocmi et al.); year and month only, 1 Nov 2021
6. Results of WMT22 Metrics Shared Task: Stop Using BLEU - Neural Metrics Are Better and More Robust — WMT 2022 (Freitag et al.); year and month only, 1 Dec 2022