How are large language models trained? Pretraining, fine-tuning and alignment

In this article

1. Start here: the learning path
2. How does pretraining work?
3. What does supervised fine-tuning do?
4. How do RLHF and reward models work?
5. What are reasoning models and GRPO?
6. What does DPO change?
7. How does LoRA make fine-tuning cheap?
8. Should you fine-tune, prompt or retrieve?
9. What is distillation, and where does it fit?
10. Questions interviewers ask on LLM training
11. All 38 guides in Fine-tuning and LLM training
12. Follow-up questions
13. Sources

Start here: the learning path

Read the stages in order, because each one starts from the weights the previous one produced. The first stage uses almost all of the compute; the later stages use far less data but decide how the model behaves.

Preference tuning

Preference tuning is the final training stage, in which the model is adjusted toward the answers people judge better, using a method such as reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO).

Pretraining

Pretraining is the first training stage of a language model, in which the model learns from a very large text collection to predict each next token (a token is a short chunk of text, often a word or part of one), and which produces a base model that continues text but does not yet follow instructions.

Supervised fine-tuning

Supervised fine-tuning (SFT) is a training stage that continues from a pre-trained model on curated pairs of prompts and ideal responses, so that the model learns to follow instructions and write in the desired format.

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF) is a training stage that fits a reward model to human rankings of model outputs and then adjusts the language model to produce outputs the reward model scores highly.

Direct preference optimization

Direct preference optimization (DPO) is a preference-tuning method that trains the language model directly on pairs of preferred and rejected responses with a classification-style loss, without a separate reward model or a reinforcement-learning loop.

Text description of the diagram

• Pretraining has historically dominated the cost, which is why almost nobody repeats it.
• Each stage starts from the weights the previous one produced.

The transformer these stages train is explained in the transformer hub, and the training mechanics (loss, gradients, optimizers) in the deep learning hub.

1. What is knowledge distillation, and how does it work?
2. What is RLHF, and how does it train a language model?
3. Fine-tuning an LLM: what it is, how it works and how much memory it needs
4. What is LoRA fine-tuning, and how is QLoRA different?
5. Anthropic Constitutional AI: how it works, step by step
6. GRPO explained: how group relative policy optimization trains LLMs
7. Fine-tuning vs prompt engineering: which should you use?
8. What is the Bradley-Terry model in RLHF reward models?
9. What is the alignment tax in an LLM, and how to reduce it?
10. DPO vs RLHF: what is the difference?
11. How do reasoning models work?
12. How do you design a post-training system for an LLM?
13. What is generative AI, and how does it work?
14. What is model tiering?
15. Why not use SFT for LLM training?
16. Can RLHF make a model honest?
17. What is RLHF, and how is it used?
18. Why do LLMs hallucinate, and how do you reduce it?
19. What is a context window in an LLM, and what does it cost?
20. How will you finetune LLMs?
21. What data strategies facilitate model learning?
22. What is BERT pretraining using MLM and NSP?
23. What does DPO optimize, and what does its loss measure?
24. How does GRPO compare to DPO?
25. Why has DPO been replacing PPO?
26. Is GRPO a policy gradient algorithm?
27. Can GRPO work with binary rewards (pass/fail)?
28. How many responses per group (G) does GRPO need?
29. Where does Constitutional AI fit?

How does pretraining work?

Pretraining trains a transformer to predict the next token of text, using the cross-entropy loss and gradient descent over a very large corpus. The model is never told what a fact or a grammar rule is; it learns whatever helps predict the next token. The result is a base model that continues a prompt in the style of its data.

Scale is the main lever, and how to split a fixed compute budget has a published answer. Hoffmann et al. (2022) trained a 70-billion-parameter model, Chinchilla, on four times as much data as the 280-billion-parameter Gopher, and argue that model size and training tokens should be scaled equally. For the same compute budget, the practical lesson is that a smaller model trained on more tokens can beat a larger one trained on fewer.

Pretraining has historically dominated the cost (even reasoning-model reinforcement learning, such as DeepSeek-R1 in 2025, is a post-training stage that starts from a pretrained base model, DeepSeek-V3-Base), which is why almost nobody repeats it. Teams start from a released base model and adapt it, which is what the next stages do.

What does supervised fine-tuning do?

Supervised fine-tuning continues training the base model on curated prompt-and-response pairs so it follows instructions instead of merely continuing text. The same recipe is called instruction tuning when the examples cover many tasks written as natural-language instructions.

Wei et al. (2021) instruction-tuned a 137-billion-parameter model on more than 60 tasks and report that the result beat zero-shot generative pre-trained transformer (GPT) 3, a 175-billion-parameter model, on 20 of 25 evaluated tasks. They name the number of fine-tuning datasets, the model scale and the use of natural-language instructions as key factors.

The loss is the same next-token loss as in pretraining (Fine-tuning an LLM walks through the full procedure), usually computed only on the response tokens. The data is the hard part: a small set of high-quality examples can matter more than a large noisy one (Zhou et al., 2023), and the format of the examples sets the format of the model's answers.

How do RLHF and reward models work?

Reinforcement learning from human feedback adds a second signal: which of several answers a person prefers. In the InstructGPT paper (2022) the recipe is labeler demonstrations for supervised fine-tuning, then reinforcement learning from human rankings of model outputs. The authors report that human evaluators preferred outputs of the 1.3-billion-parameter InstructGPT, which has over 100 times fewer parameters, to those of the 175-billion-parameter GPT-3, with improved truthfulness and less toxic output.

The pieces are a reward model, which is a network trained to score an answer the way human rankings would, and a policy update (the policy is the language model being trained, viewed as a rule for choosing the next token) that raises the probability of answers the reward model scores highly while keeping the model close to its starting point. The cost is complexity: up to four models in memory (the policy, a frozen reference copy for the penalty that keeps it close to its start, the reward model and a value model), a sampling loop, and a reward model that the policy can learn to exploit.

What are reasoning models and GRPO?

A reasoning model is a language model that, after pretraining, is trained mostly with reinforcement learning on checkable problems to write out intermediate steps before its final answer. Reinforcement learning also works without human rankings when an answer can be checked by a program, such as a maths result or passing unit tests. This is reinforcement learning with verifiable rewards: the reward is 1 for a correct answer and 0 otherwise, so no learned reward model has to score it.

Shao et al. (2024) introduced group relative policy optimization (GRPO) in DeepSeekMath, which samples a group of answers per prompt and scores each answer by how far it sits above or below the group's average, divided by the group's spread (standard deviation), instead of training a separate value model. DeepSeekMath itself scored answers with a trained reward model; DeepSeek-AI (2025) later paired GRPO with rule-based accuracy and format rewards to train DeepSeek-R1-Zero and the reasoning stages of DeepSeek-R1; the final stage of R1 also used learned reward models for general prompts. The step-by-step version is on the GRPO page.

What does DPO change?

Direct preference optimization removes the separate reward model and the reinforcement-learning loop. Rafailov et al. (2023) reparameterise the RLHF objective so it can be solved by training on preference pairs with a simple classification loss, and report that it matches or beats the reinforcement-learning approach on sentiment control, summarisation and dialogue while being simpler to implement and train.

The trade-off interviewers want named: DPO is simpler and more stable to run, but it learns from a fixed set of preference pairs, whereas the reinforcement-learning route can sample fresh answers from the current model. Which is better depends on the data you have and the compute you can spend, and the published comparison covers only the tasks the authors tested.

How does LoRA make fine-tuning cheap?

Low-rank adaptation (LoRA) freezes the pre-trained weights and trains two thin matrices beside each chosen weight matrix; the update is called low-rank because it is built from those two thin matrices instead of one full one. Hu et al. (2021) report a 10,000-times cut in trainable parameters and a 3-times cut in graphics processing unit (GPU) memory for the 175-billion-parameter GPT-3 model against Adam fine-tuning. Dettmers et al. (2023) add a frozen 4-bit base (QLoRA is quantized LoRA: LoRA on a base whose numbers are stored in fewer bits, called quantization) and fine-tune a 65-billion-parameter model on one 48 GB GPU.

The memory arithmetic is the part to remember. Full fine-tuning with Adam in mixed precision needs 16 bytes per parameter for weights, gradients and optimizer state kept in mixed precision (Rajbhandari et al., 2019). For a 7B model, about 6.7 billion parameters, that is 107.8 gigabytes (GB), against 13.5 GB with LoRA and about 3.4 GB with a 4-bit base, all before activations, the intermediate values kept for the backward pass, so real runs need more than these figures.¹ The full walk-through with parameter counts is on the LoRA and QLoRA page.

Text description of the diagram

• For a 7B model that is 107.8 gigabytes against 13.5 GB with LoRA and about 3.4 GB with a 4-bit base.
• Activations are excluded, so real runs need more than these figures.

Should you fine-tune, prompt or retrieve?

Fine-tuning changes the model's weights (fine-tuning versus prompt engineering compares them in depth); prompting and retrieval leave them alone. Fine-tuning is the right tool when you need a consistent style, format or skill that a prompt cannot reliably produce. Retrieval-augmented generation, introduced by Lewis et al. (2020), pairs a pre-trained model with an index of documents it can search, which suits knowledge that changes or that must be cited.

A fine-tune bakes knowledge into weights at training time, so updating it means training again; a retrieval index can be updated by adding a document. A common answer is to start with the prompt, add retrieval for facts, and fine-tune only when behaviour still falls short. The inference side of these choices, including how a trained model is served quickly, is covered in the LLM inference hub, and production operation in the machine learning operations hub.

What is distillation, and where does it fit?

Distillation trains a small model to imitate a larger one. Hinton, Vinyals and Dean (2015) show how to compress the knowledge of an ensemble into a single model by training the small model on the large model's output probabilities instead of only the hard labels. The large model is the teacher and the small one the student; the soft probabilities carry more information per example than a single right-or-wrong label, such as which wrong answers are nearly right.

Distillation therefore needs the teacher's outputs but not its weights, and the student can be many times smaller. It sits alongside fine-tuning as a way to get a cheaper model for serving. For the number formats that make serving cheaper, see 16-bit number formats and GPU fundamentals.

Questions interviewers ask on LLM training

Four questions open most rounds on this topic.

1. RLHF or DPO for a new chat model? RLHF trains a reward model and then optimises against it; DPO optimises directly on preference pairs without one, so it is simpler to run. See DPO versus RLHF.
2. How much memory does full fine-tuning of a 7B model need? About 108 GB of model state at 16 bytes per parameter, before activations.¹
3. Why is supervised fine-tuning not enough? It teaches the format of good answers but not which of several plausible answers people prefer; preference tuning supplies that signal.
4. Why use a reward model? It turns human rankings into a score the training loop can optimise, at the cost of a model the policy can exploit.

Follow-up questions

What are the stages of training a large language model?

Pretraining on a large text corpus, supervised fine-tuning on prompt-and-response pairs, then preference tuning with a method such as RLHF or DPO. Each stage starts from the previous stage's weights; Fine-tuning an LLM walks through the procedure.

What is the difference between pretraining and fine-tuning?

Pretraining learns general language ability from very large unlabeled text and takes most of the compute. Fine-tuning continues from those weights on a much smaller curated dataset to change behaviour or add a skill, as described in Fine-tuning an LLM.

When should you fine-tune instead of prompting or retrieving?

Fine-tune when a prompt cannot reliably produce the style, format or skill you need, and retrieve when the model needs facts that change or must be cited. Most teams try prompting first because it needs no training; see fine-tuning versus prompt engineering.

What is the difference between RLHF and DPO?

RLHF trains a reward model on human rankings and then optimises the language model against it with reinforcement learning. DPO trains directly on preference pairs with a classification loss and skips both the reward model and the loop. The DPO versus RLHF page compares them.

Why not stop after supervised fine-tuning?

Supervised fine-tuning teaches the format of good answers from one example per prompt, but it cannot say which of several plausible answers people prefer. Preference tuning supplies that signal through RLHF or DPO.

What is perplexity?

Perplexity is the exponential of the mean cross-entropy loss per token on a text sample. A perplexity of 10 means the model is, on average, as uncertain as if it were choosing among 10 equally likely tokens.

How much data per parameter should pretraining use?

Hoffmann et al. (2022) trained Chinchilla with 70 billion parameters on 1.4 trillion tokens, which is 1.4 trillion divided by 70 billion, or 20 tokens per parameter, and argue for scaling parameters and tokens in equal proportion (see how pretraining works). Treat 20 as the figure for that setting, not a universal law.

Why does fine-tuning need far less memory with LoRA?

Because only small adapter matrices train, so gradients and optimizer state exist only for them. In the 7B example the model-state memory falls from 107.8 GB to 13.5 GB.¹ The method itself is on the LoRA and QLoRA page.

Sources

1. Training language models to follow instructions with human feedback — arXiv (Ouyang et al.), 4 Mar 2022
2. Finetuned Language Models Are Zero-Shot Learners — arXiv (Wei et al.), 3 Sep 2021
3. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — arXiv (Rafailov et al.), 29 May 2023
4. Training Compute-Optimal Large Language Models — arXiv (Hoffmann et al.), 29 Mar 2022
5. LoRA: Low-Rank Adaptation of Large Language Models — arXiv (Hu et al.), 17 Jun 2021
6. QLoRA: Efficient Finetuning of Quantized LLMs — arXiv (Dettmers et al.), 23 May 2023
7. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — arXiv (Rajbhandari et al.), 4 Oct 2019
8. Distilling the Knowledge in a Neural Network — arXiv (Hinton, Vinyals, Dean), 9 Mar 2015
9. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — arXiv (Shao et al.), 5 Feb 2024
10. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — arXiv (DeepSeek-AI), 22 Jan 2025
11. LIMA: Less Is More for Alignment — arXiv (Zhou et al.), 18 May 2023
12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv (Lewis et al.), 22 May 2020