Fine-Tuning & LLM Training

Why not use SFT for LLM training?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 10 min read

Why not use supervised fine-tuning (SFT) for large language model (LLM) training? You can, and it is a strong first stage, but supervised fine-tuning (SFT) only imitates demonstrations: it teaches the format of a good answer, gives no signal for what to avoid, and cannot rank two good answers. The follow-up is why flawless demonstrations still leave a model drifting: with an illustrative 1% slip per token, only 13% of 200-token answers stay on track.¹

In this article

Before you start
Why interviewers ask this
SFT from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page sits on the LLM training learning path, after how transformers work.

Why interviewers ask this

The question separates candidates who know that alignment is a set of stages from those who treat "fine-tuning" as one step. A weak answer says "SFT is outdated" or "SFT is worse than RLHF". A strong answer says what SFT optimizes (imitation of one reference answer per prompt), what that leaves out (a signal for bad answers and for ranking good ones) and what it is still good at. A Staff-level answer adds the evidence on both sides: Zhou et al. (2023) show how far a small SFT set goes, and Ouyang et al. (2022) show where human-feedback training pulls ahead.

SFT from first principles

Teacher forcing: Teacher forcing is a way of computing the SFT loss in which the model is always shown the correct previous tokens of the reference response, never its own earlier guesses, while it predicts each next token.

In plain words

Think of learning to cook by watching a chef and copying every move. You learn the style fast. You never learn which of two dishes the diner would have preferred, because you only ever saw one dish per order. The analogy breaks in one place: a model copying a chef also copies tokens it has no reason to believe.

In one precise sentence, SFT minimizes the cross-entropy between the model's next-token distribution and the reference response, token by token.

Step 1: SFT trains the model to imitate demonstrations

Demonstrations teach a model to imitate the next token. The input is a prompt and a reference answer. At each position the model predicts the next token given the correct prefix, which is teacher forcing, and the loss is the negative log of the probability it gave the reference token.

Take a five-token reference answer where the model gives the correct tokens probabilities 0.60, 0.30, 0.50, 0.20 and 0.80 (illustrative). The per-token losses are 0.5108, 1.2040, 0.6931, 1.6094 and 0.2231, the total is 4.2405 and the mean is 0.8481.¹ The probability of the whole reference answer is 0.0144, and minus its natural log is again 4.2405.¹ You now know what SFT optimizes: raise the probability of this one answer, token by token.

Step 2: SFT mostly teaches format and style, not new knowledge

SFT mostly teaches the format and style of an answer. Zhou et al. (2023) tuned a 65B-parameter LLaMa model with the standard supervised loss on only 1,000 carefully curated prompts and responses, with no reinforcement learning or human preference modeling. In their human study, responses from that model, called Less Is More for Alignment (LIMA), were equivalent or preferred to Generative Pre-trained Transformer 4 (generative pre-trained transformer (GPT)-4) in 43% of cases.

Their explanation is the Superficial Alignment Hypothesis: knowledge and capabilities are learned almost entirely during pretraining, while alignment teaches which format to use with users. It is a hypothesis, not a settled fact. If a demonstration states a fact the model does not hold, the loss still demands it, so SFT is an unreliable way to add knowledge; Gekhman et al. (2024) study whether this encourages hallucinations. See fine-tuning vs prompt engineering for when each tool fits.

Step 3: SFT has no negative signal and cannot rank answers

SFT gives no signal about which of two answers is better. It sees one reference per prompt and never sees the model's own bad answers, so it cannot say "this plausible response is worse than that one". Raising one reference's probability lowers other tokens only as a side effect of the softmax.

SFT can at best match its demonstrations. Comparing two answers is easier for a labeler than writing the ideal one, which is why Ouyang et al. (2022) collect rankings of model outputs after the demonstration stage.

Step 4: Why does a model trained on perfect demonstrations still drift?

A model trained on perfect demonstrations still drifts because teacher forcing shows it only clean prefixes while at use time it reads its own earlier tokens. Training uses perfect prefixes and use runs on the model's own tokens. Suppose each generated token has an illustrative 1% chance of leaving the territory the demonstrations covered. The chance of staying on track is 0.9044 over 10 tokens, 0.605 over 50 and 0.134 over 200.¹ A small error rate per token compounds.

This gap is sometimes called exposure bias, and how much it matters for large language models is debated. During training the model never recovers from its own slip. At use time one odd token changes every later prediction. An on-policy preference stage, which trains on answers sampled from the current model, sees those slips and can mark them as worse than a good alternative.

A small per-token drift compounds over a long response, which is why imitation alone is brittle.CC BY 4.0 — source: Why not use SFT for LLM training?

Text description of the diagram

The chance of staying on track is 0.9044 over 10 tokens.
The chance of staying on track is 0.605 over 50 tokens and 0.134 over 200.

Step 5: Preference tuning adds the missing comparison signal

Preference tuning learns from comparisons between answers. Instead of one reference, each example is a preference pair, and the loss rewards the better answer scoring above the worse one. The two best-known routes are reinforcement learning from human feedback (RLHF), covered in what RLHF is and driven by a reward function, and direct preference optimization (DPO), which Rafailov et al. (2023) present as solving the RLHF problem with a simple classification loss; the DPO vs RLHF comparison covers the difference.

DPO learns from a fixed set of pairs unless they are sampled from the current model. The pairwise loss is $- \log σ (m)$ , where $σ$ is the sigmoid function and m is the better answer's score minus the worse one's. With an illustrative margin of 0.8 it is 0.3711; with no margin it is 0.6931.¹ Ouyang et al. (2022) report that human raters preferred outputs from the 1.3B-parameter InstructGPT model to those of the 175B-parameter Generative Pre-trained Transformer 3 (GPT-3), despite 100 times fewer parameters. Chu et al. (2025) report that SFT tends to memorize while outcome-reward reinforcement learning (RL) generalizes better, and that SFT still helps by stabilizing the output format first.

SFT imitates demonstrations; the missing comparison signal is what preference tuning adds.CC BY 4.0 — source: Why not use SFT for LLM training?

Step-by-step description

Demonstrations teach next-token imitation.
SFT mostly teaches format and style.
No signal about which answer is better.
Perfect prefixes in training, own tokens at use.
Preference tuning learns from comparisons.

Demonstrations teach a model to imitate the next token.
SFT mostly teaches the format and style of an answer.
SFT gives no signal about which of two answers is better.
Training uses perfect prefixes and use runs on the model's own tokens.
Preference tuning learns from comparisons between answers.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.