In this article
Before you start
This page is one step of the generative AI system design learning path, so it stays at the level a design round needs: what each stage does, in what order, and why.
Why interviewers ask this
Training questions test whether you know where a model's behaviour comes from. A design round on a chat product, such as designing ChatGPT or Claude.ai, can reach "how was the model made?" in a single follow-up, and the answer decides what you can fix with a prompt and what needs new training.
A weak answer says "you feed it lots of text and it learns." A strong answer separates two stages with different goals, names the data each one needs, and explains why a pretrained model is not yet an assistant. A Staff-level answer adds the economics: the model size and token count are chosen together under a compute budget, the later stages are cheap compared with pre-training, and the whole thing is wrapped in evaluation gates before release.
LLM training pipeline from first principles
- LLM training pipeline
- An LLM training pipeline is the ordered set of stages, from data preparation through pre-training and post-training to evaluation, that turns raw text into a language model able to follow instructions, with each stage consuming the previous stage's output.
- Pre-training
- Pre-training is the first and most expensive stage of building a large language model, in which the network learns to predict the next token over a huge text corpus, which gives it broad knowledge of language but not yet assistant behaviour.
- Post-training
- Post-training is the set of later stages, such as supervised fine-tuning (SFT) and preference tuning, that adjust a pretrained model so it follows instructions and matches human preferences, using far less data than pre-training.
In plain words
Think of training a new employee. First they read an enormous library, which gives them knowledge and a feel for language but no job skills. Then a mentor shows them worked examples of good replies. Then the mentor ranks several attempts and the employee learns which kind of reply people prefer. The analogy breaks in one place: a model does not choose to read or to improve, an optimisation procedure changes its numbers whenever its predictions are wrong.
In one precise sentence, an LLM training pipeline first minimises next-token prediction error on a large corpus, then shifts the model's behaviour with smaller, curated datasets of demonstrations and preferences.
Step 1: Data is turned into tokens
A token is a small piece of text, such as a word or part of a word, that the model reads and predicts as one unit. The first stage converts a very large text collection into tokens. The Llama 3 report (Meta AI, 2024) starts by converting a large, multilingual text corpus into discrete tokens and then pre-trains a model on the result.
Input: raw documents. Output: a stream of token ids. Before that conversion, teams clean and filter the text; the Llama 3 report describes pre-processing and curation pipelines for its data. The quality of this stage caps everything after it, because no later stage can teach facts the model never saw. You now know what the model will read; the next step decides how much of it and how big the reader is.
Step 2: Pre-training sets model size and token count
Pre-training predicts the next token on a huge text corpus. Each prediction error nudges the model's parameters, and repeating this over trillions of tokens produces a model with a rich understanding of language that does not yet behave like an assistant.
Two numbers define the run: parameters and tokens. Llama 3's flagship has 405B parameters trained on 15.6T tokens, about 38.5 tokens per parameter.1 A scaling law is an empirical formula that predicts how a model's loss changes with parameters, data and compute. Hoffmann et al. (2022) find that for compute-optimal training, model size and token count should scale equally: for every doubling of model size, the tokens should also double.
To judge whether 38.5 is high or low, use their Chinchilla model as the reference: 70B parameters trained on 1.4T tokens, which is 20 tokens per parameter.1 The Llama 3 flagship therefore sits at almost twice the Chinchilla ratio, about 1.9 times.1
Step 3: Supervised tuning teaches the format of an answer
Post-training tunes the model to follow instructions and align with human preferences. It exists because the pretraining objective is misaligned with what users want. Ouyang et al. (2022) write: "Making language models bigger does not inherently make them better at following a user's intent."
The first post-training stage is supervised fine-tuning (SFT). Labellers write demonstrations of the desired behaviour, and the model is trained on them with ordinary supervised learning: given this prompt, produce this reply. The loss is the same next-token loss as before, applied to a small, carefully written dataset.
Input: prompts paired with good replies. Output: a model that answers in the expected format instead of continuing the prompt like a web page. SFT teaches the shape of a good answer, but demonstrations cannot cover every judgement call, which is why a second signal is added next.
Step 4: Preference tuning teaches which answer people prefer
People find it easier to rank two answers than to write a perfect one. Ouyang et al. (2022) collect rankings of model outputs and use them to tune the supervised model further with reinforcement learning from human feedback (RLHF). A reward model is a network trained on those rankings that scores a reply by how much people would prefer it, and a reinforcement learning algorithm such as proximal policy optimization (PPO) then pushes the model toward higher-scoring replies.
Direct preference optimization (DPO) removes two of those parts. Rafailov et al. (2023) show the objective can be optimised in a single stage, essentially solving a classification problem on the preference data. DPO trains on preference pairs in a single stage with no separate reward model.
Input: pairs of a preferred and a rejected reply. Output: a model whose outputs match what raters preferred.
Real recipes mix the methods. The Llama 3 report (Meta AI, 2024) describes several rounds of post-training that combine supervised fine-tuning, rejection sampling and DPO, rather than PPO-based RLHF.
Step 5: Evaluation gates the release, and continuous training repeats the loop
Before release, the model is scored on held-out tests and compared with the previous version; a regression blocks the release. Those tests typically cover capability benchmarks, instruction following and safety behaviour, and the comparison with the previous version shows whether the new model is actually better. Evaluation is also where you find which stage to repeat, because a failure on instruction following points to supervised tuning, while a failure on style or safety points to preference tuning. The same habit applies to retrieval systems, see evaluating a retrieval-augmented generation (RAG) system stage by stage.
Machine learning operations (MLOps) is the engineering practice of unifying the development and the operation of machine learning systems. Continuous training (CT) is the MLOps practice of automatically retraining and redeploying models through a multi-step pipeline, as described in Google Cloud's MLOps architecture guide. The next section applies it to an LLM.
How do you design a continuous training pipeline for an LLM?
A continuous training pipeline for an LLM is an automated loop that retrains the lightweight stages, validates the result and releases it only if the checks pass. Google Cloud's MLOps guide lists five retraining triggers: on demand, on a schedule, when new training data arrives, when model performance degrades and when the data distribution changes significantly. It puts automated data validation before training and automated model validation before deployment. The same guide describes maturity levels 0, 1 and 2: manual, automated training pipeline, and automated build, test and deployment of the pipeline itself.
Because the guide targets predictive systems, the LLM mapping is an inference: new preference data or a measured quality drop is the trigger, the retrained step is usually supervised or preference tuning on refreshed data, validation is a regression evaluation gate against the current model, and release is a canary that sends a small share of traffic to the new version first. The lightweight post-training stages fit this retrain-and-gate loop far better than a full pre-training run does.
The LLM training pipeline explained in one diagram
The LLM training pipeline explained in one line: data becomes tokens, pre-training learns language, supervised and preference tuning teach behaviour, and evaluation gates the release. The diagram below shows the five stages in order. Data hands token ids to pre-training, pre-training hands base weights to supervised tuning, supervised tuning hands a supervised checkpoint to preference tuning, and preference tuning hands a preference-tuned checkpoint to evaluation. Each box feeds the next, so a change to an early stage, such as new data, means every later stage must be rerun or at least re-checked before a new model is released to users.
Step-by-step description
- Raw text becomes tokens.
- The model learns next-token prediction.
- Demonstrations teach the answer format.
- Rankings teach which answer is preferred.
- Held-out tests gate the release.
- Pre-training predicts the next token on a huge text corpus.
- Post-training tunes the model to follow instructions and align with human preferences.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take the sourced Llama 3 figures and check them by hand. The report states a flagship of 405B parameters, 15.6T tokens and 3.8e25 floating-point operations (FLOPs) of compute. A common approximation from the scaling-law literature says training compute is about 6 times parameters times tokens, which is an approximation and not a result of the Llama 3 report. It gives 6 × 405e9 × 15.6e12 ≈ 3.79e25, within about 0.25% of the reported value.1
Llama 3 was pre-trained on about 15T tokens, compared to 1.8T tokens for Llama 2, per Meta AI (2024), a growth of about 8.3 times.1
| Quantity | Value |
|---|---|
| Compute from 6 × parameters × tokens, Llama 3 flagship | 3.79e25 FLOPs |
| Compute stated in the Llama 3 report | 3.8e25 FLOPs |
| Tokens per parameter, Llama 3 flagship | 38.5 |
| Tokens per parameter, Chinchilla (70B parameters, 1.4T tokens) | 20 |
| Llama 3 flagship ratio against the Chinchilla ratio | about 1.9 times |
| Token growth, Llama 2 to Llama 3 | 8.3 times |
| Chinchilla parameters vs Gopher (70B vs 280B) | 4 times fewer |
| InstructGPT parameters vs generative pre-trained transformer (GPT) 3 (1.3B vs 175B) | about 135 times fewer1 |
| Compute when both model size and tokens double | 4 times |
The last row follows from the approximation: doubling parameters and doubling tokens multiplies their product by four, so equal scaling is expensive.1 The Chinchilla model uses the same compute budget as Gopher with 70B parameters and 4× more data, so the saving is a smaller model that is cheaper to run, not a cheaper training run. Ouyang et al. (2022) report that outputs from a 1.3B InstructGPT model are preferred to those from the 175B GPT-3, a model about 135 times larger, which is how post-training can beat raw scale.1
Text description of the diagram
- Llama 3 was pre-trained on about 15T tokens, compared to 1.8T tokens for Llama 2.
- That is a growth of about 8.3 times.
Trade-offs and edge cases
The main design choice is how to spend a fixed budget. A bigger model costs more at every request, so a team that expects heavy use may train a smaller model on more tokens than the compute-optimal point. The Llama 3 report says its flagship is approximately compute-optimal, while its smaller models are trained for much longer than is compute-optimal. The second choice is how to learn from preferences. RLHF fits a reward model first and then runs reinforcement learning. DPO skips both. Rafailov et al. (2023) report that, with virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO. Neither removes the need for preference data.
As of 2025, reasoning models add another route: reinforcement learning with rule-based, verifiable rewards. DeepSeek-R1 uses group relative policy optimization (GRPO) in this way (DeepSeek-AI, 2025), as a reinforcement-learning stage added to the pipeline.
Text description of the diagram
- RLHF fits a reward model first and then runs reinforcement learning.
- DPO trains on preference pairs in a single stage with no separate reward model.
Edge cases: pre-training cannot be patched later with a prompt, so missing knowledge needs new data or retrieval (see do you even need RAG?); the human rankings reflect the raters' judgement, not a universal truth; and the Google MLOps guide targets predictive systems, so its retraining triggers need adapting before they govern an LLM.
How to answer it in an interview
Walk me through how an LLM is trained.
There are two phases. Pre-training predicts the next token over a huge text corpus, and the model size and token count are chosen together under a compute budget. That gives knowledge, not an assistant. Post-training fixes that: supervised tuning on written demonstrations, then preference tuning, either reinforcement learning from human feedback with a reward model or direct preference optimization, which skips the reward model. I would gate each release on held-out evaluations. The trade-off is cost: as of 2025 pre-training is usually still the biggest bill, though reinforcement learning for reasoning models adds a further stage.
The spoken structure: name the two phases, give each one's data and goal, name the preference-tuning choice, then close with evaluation and cost.
Common mistakes
- "A pretrained model is already a chatbot." (wrong) It predicts the next token and does not yet follow instructions as an assistant should. Say instead: "Pre-training gives language ability; post-training gives assistant behaviour." (right)
- "Bigger is always better." (wrong) On the authors' prompt distribution, labellers preferred a 1.3B tuned model over 175B GPT-3. Say instead: "Alignment and data matter as much as size." (right)
- "Chinchilla means one fixed token ratio for every model." (wrong) Llama 3 trains small models far past compute-optimal on purpose. Say instead: "Compute-optimal is a starting point; serving cost can justify more tokens." (right)
- "DPO removes the need for human data." (wrong) It removes the reward model and the reinforcement learning loop only. Say instead: "DPO still trains on preference pairs." (right)
Follow-up questions
What is the difference between pre-training and post-training?
Pre-training learns language by predicting the next token over a huge corpus and costs most of the compute. Post-training uses much smaller datasets of demonstrations and preferences to make the model follow instructions and match human preferences.
Is a pretrained LLM already a chatbot?
No. The Llama 3 report says the pretrained model has a rich understanding of language but does not yet follow instructions or behave the way an assistant should, and Ouyang et al. call the language-modelling objective misaligned for that reason.
What is the difference between RLHF and DPO?
RLHF trains a reward model on rankings, then optimises the language model against it with reinforcement learning. DPO optimises the preference objective directly with a cross-entropy-style loss, so it needs no separate reward model. Both need preference data.
Does Chinchilla mean every model should be trained to one fixed ratio?
No. It says that for a fixed compute budget, parameters and tokens should scale together. Teams may still train smaller models for longer, because a smaller model is cheaper to serve.
What does MLOps continuous training pipeline design mean for an LLM?
It means automating retraining, evaluation and redeployment as one loop, as set out in the section above on designing a continuous training pipeline for an LLM; platform questions such as the LLM gateway sit around it.
- The LLM training pipeline runs data, pre-training, supervised tuning, preference tuning and evaluation, in that order.
- Pre-training teaches language by next-token prediction; post-training teaches assistant behaviour.
- Model size and token count are chosen together under a compute budget, and serving cost can justify going past compute-optimal.
- DPO replaces the reward model and reinforcement learning loop but still needs preference data.
- Continuous training automates retrain-evaluate-release, and for an LLM it fits post-training best.
Sources
- Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv / NeurIPS 2022 (Ouyang et al.), 4 Mar 2022
- The Llama 3 Herd of Models (arXiv 2407.21783) — Meta AI (arXiv), 31 Jul 2024
- Training Compute-Optimal Large Language Models (arXiv 2203.15556) — arXiv / NeurIPS 2022 (Hoffmann et al.), 29 Mar 2022
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv / NeurIPS 2023 (Rafailov et al.), 29 May 2023
- MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center, 28 Aug 2024
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025
