What is BERT pretraining using MLM and NSP?

In this article

1. Before you start
2. Why interviewers ask this
3. BERT pretraining from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the how large language models are trained learning path.

Why interviewers ask this

Saying that BERT predicts the next word ends the answer early, because that describes a different kind of model. The question separates candidates who know how an encoder learns from text without labels from those who repeat two acronyms. A weak answer names MLM and NSP and stops. A strong answer explains what each task hides, what the model must recover, and why the hiding is needed for bidirectional training. A Staff-level answer adds what changed afterwards: a successor model dropped the sentence-pair task, and the two papers disagree on whether it helps. The question is reported by candidates in interviews on language-model fundamentals; no public source ties it to a named company.

BERT pretraining from first principles

Masked language modelling

Masked language modelling (MLM) is a self-supervised training task that hides or replaces a small share of the tokens in a sequence and asks the model to recover the original tokens from the text around them, which lets a model learn from raw text because the text itself supplies the answers, with no human labels.

Next sentence prediction

Next sentence prediction (NSP) is a two-way classification task that gives the model two text segments, A and B, and asks whether B really followed A in the source text or was swapped in from somewhere else, so the model is pushed to represent how two pieces of text relate to each other, not only the words inside each one.

Bidirectional context

Bidirectional context is the use of the tokens on both the left and the right of a position when the model builds that position's representation, in contrast to a left-to-right model, which sees only earlier tokens; the BERT paper describes jointly conditioning on both sides in all layers.

In plain words

Think of a fill-in-the-blank worksheet cut from a million books. A student who does thousands of them must learn grammar, facts and word meaning. Nobody has to mark the answers, because the books hold them. MLM is that worksheet. NSP is a second worksheet that shows two passages and asks whether the second came right after the first.

The analogy breaks in one place. A student learns rules in words, while the model learns numbers that stand for each token.

An encoder is a network that turns a sequence of tokens into one vector for each token, using the whole sequence. Pretraining is the first, general training stage in which a model learns from unlabelled text before any task-specific adaptation. In one precise sentence, BERT pretraining trains a transformer encoder to solve the two tasks at once, using only unlabelled text.

Step 1: Masked language modelling hides tokens and the model recovers them

The model predicts masked tokens from both sides of each gap, and the paper's goal is "jointly conditioning on both left and right context in all layers" (Devlin et al., 2018, revised 2019).

Devlin et al. (2019), Section 3.1 choose 15% of the token positions at random for prediction. A chosen token becomes the [MASK] symbol 80% of the time, a random token 10% of the time and stays unchanged 10% of the time. The mix matters because the mask symbol never appears at fine-tuning time.

Take an illustrative sentence of 20 tokens: 15% selects 3 tokens.¹ Two become the mask symbol and one a random word.

The loss counts only the 3 selected positions, and the other 17 add nothing.¹ With illustrative probabilities of 0.60, 0.25 and 0.05 for the true tokens sat, dog and door, the losses are 0.5108, 1.3863 and 2.9957. The masked token door costs 2.9957 because the model gave it the lowest probability; the mean loss is 1.631.¹

Text description of the diagram

• The loss of the masked token door is 2.9957.
• The mean loss over the three masked tokens is 1.631.

Step 2: Next sentence prediction labels sentence pairs, and RoBERTa dropped it

Next sentence prediction asks whether sentence B follows sentence A. Each example packs two segments into one input, with a classification token first and a separator token after each segment. In Devlin et al. (2019), Section 3.1, 50% of the time B is the actual next sentence and 50% of the time it is a random sentence from the corpus. The aim is to help question answering and natural language inference.

In the illustrative example, a true pair gets probability 0.9 for "follows" and a random pair gets 0.2, so the two losses are 0.1054 and 0.2231, a mean of 0.1643.¹

The two papers disagree on whether NSP helps. The ablation in Devlin et al. (2019), Table 5 reports that removing it hurt Question-answering Natural Language Inference (QNLI), Multi-Genre Natural Language Inference (MNLI) and Stanford Question Answering Dataset (SQuAD) results. Liu et al. (2019), Section 4.2 find that removing the NSP loss "matches or slightly improves downstream task performance". Lan et al. (2019) argue NSP is too easy because random segments differ in topic, and replace it with sentence-order prediction.

Step 3: Both losses train one encoder, then a small output layer adapts it

The two losses are added into one training signal, so a single backward pass updates every weight of the encoder. With the illustrative numbers, the mean masking loss of 1.631 plus the mean sentence-pair loss of 0.1643 gives 1.7952.¹

A masked-token head reads the encoder output at each gap, and a sentence head reads the first token's output. Pretraining ends with a general encoder, not a finished tool. Devlin et al. report that the pretrained model can be adapted with just one additional output layer. That adaptation is fine-tuning a pretrained model in its simplest form, and parameter-efficient fine-tuning (PEFT) updates only a small added part.

1 · Mask tokens2 · Pair sentences3 · One encoder, one lossReplay

Step-by-step description

1. Some tokens are hidden and the model predicts them from both sides.
2. Two segments are paired and the model says if B follows A.
3. The two losses are added into one training signal.

• The model predicts masked tokens from both sides.
• Next sentence prediction asks whether sentence B follows sentence A.
• The two losses are added into one training signal.

The Staff-level answer

What makes it Staff-level: it explains why the mask is mixed with random and unchanged tokens, it ends with the successor evidence, and it treats NSP as a design choice whose value the two papers dispute. The spoken structure is: define the encoder, give MLM, give NSP, then say what changed.

The rubric

Three variants of this question

Does BERT predict the next word?

No. BERT is an encoder that recovers masked tokens using both sides of the context, so it is not a left-to-right text generator. NSP gives a yes-or-no label for a sentence pair, which is not a next-word prediction either.

Why does BERT mask tokens instead of predicting the next token?

When every position sees the whole sequence in every layer, asking a position for its own token would be trivial, because it could copy it. Hiding the token turns the task into recovery from the surroundings, and a left-to-right target would waste the right-hand context.

What happens if you remove NSP from BERT pretraining?

The two papers give different answers. Devlin et al. (2019), Table 5 report that removing NSP lowered QNLI, MNLI and SQuAD results. Liu et al. (2019), Section 4.2 find that removing the NSP loss "matches or slightly improves downstream task performance". Say the value is contested and that later recipes do without it.

Common mistakes

• Wrong: "BERT predicts the next word." Why: BERT recovers masked tokens from both sides, and it is an encoder. Right: "It fills gaps using left and right context."
• Wrong: "Every selected token becomes the mask symbol." Why: 80% do, 10% become a random token and 10% stay unchanged (Devlin et al., Section 3.1). Right: "Most become the mask symbol, some a random token and some stay unchanged, so the model does not depend on the mask."
• Wrong: "The loss covers every token." Why: in the toy example only 3 of 20 positions count.¹ Right: "The loss is taken at the selected positions."
• Wrong: "NSP is what teaches BERT sentence meaning." Why: NSP is one task, RoBERTa trains without it, and the two papers disagree on its value. Right: "NSP is one design choice with contested value."
• Wrong: "Pretraining already solves my task." Why: the pretrained model is adapted, for example with one added output layer. Right: "Pretraining gives the general encoder; fine-tuning specialises it."

Follow-up questions

How is a pretrained BERT used for a downstream task?

You add one small output layer and train on labelled examples, as Devlin et al. describe. Whether to do that or to prompt a generative model instead is the question of fine-tuning vs prompt engineering.

What does RoBERTa change in BERT pretraining?

Liu et al. (2019) say BERT "was significantly undertrained" and train with dynamic masking, full sentences without an NSP loss, and large mini-batches. They describe the original BERT budget (Section 2.6) as 1,000,000 updates of 256 sequences of at most 512 tokens, so at most 131,072,000,000 token positions.¹

Is BERT a large language model?

BERT is a pretrained encoder-only transformer, not a generative chat model. Devlin et al. (Section 3) report two sizes: BERT-base with 110 million parameters and BERT-large with 340 million. It is a pretrained language model, much smaller than most current generative ones.

How would you adapt BERT cheaply?

Use a parameter-efficient method that trains a small number of added weights and leaves the pretrained ones frozen, such as low-rank adaptation (LoRA) fine-tuning. Gradients, optimizer state and the saved checkpoint then scale with the added weights; the frozen encoder is still loaded and run in full.

Sources

1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) — arXiv (Devlin, Chang, Lee, Toutanova), 24 May 2019
2. RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv 1907.11692) — arXiv (Liu et al.), 26 Jul 2019
3. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (arXiv 1909.11942) — arXiv (Lan et al.), 26 Sep 2019