How do you design a post-training system for an LLM?

In this article

1. Before you start
2. Why interviewers ask this
3. A post-training system from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step in the large language model (LLM) training path: pretraining, fine-tuning and alignment.

Why interviewers ask this

A design question on post-training tests whether you can turn a vague goal, such as "make the model a better assistant", into data, training stages, a feedback signal and a release gate. Candidates have reported variants in interviews at AI labs and data-labeling companies: a pipeline for a coding model, an asynchronous system for a chat model already serving traffic, and noisy preference labels for a large language model (LLM).

A weak answer says "supervised fine-tuning, then reinforcement learning from human feedback (RLHF)" and stops. A strong answer says where each stage's signal comes from and how it is checked. A Staff-level answer adds the failure modes: noisy labels, stale samples, a reward that gives no contrast, and a model that gets worse on something nobody measured.

A post-training system from first principles

Post-training system

A post-training system is the set of data pipelines, training stages, scoring services and evaluations that turn a pretrained language model into a model that follows instructions, and that decide whether the result may ship.

Rollout

A rollout is a complete output sampled from the model being trained, together with its score, that lets the learner raise the probability of high-scoring outputs and lower that of low-scoring ones.

In plain words

Picture a student who has read an entire library but never sat an exam. Post-training is the tutoring: showing worked answers, ranking the student's attempts, and marking tests that have right answers. The analogy breaks in two places. One model serves every request with the same weights, so a mistake is copied at scale. And a student can learn to please the marker rather than the subject, which a model also does when the scoring is weak.

In one precise sentence, a post-training system feeds a pretrained model examples and scores, in stages, and releases the result only if held-out evaluations pass.

Ouyang et al. (2022) describe an early recipe: collect labeler demonstrations and fine-tune with supervised learning, then collect rankings of model outputs and fine-tune further with reinforcement learning from human feedback. Lambert et al. (2024, v5 2025) use supervised fine-tuning, Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR). Post-training runs in five steps: data, fine-tuning, a training signal, a training loop and a release gate.

Step 1: Collect prompts, demonstrations and preference pairs

The data step builds three sets from one prompt pool: demonstrations (a prompt and an ideal answer), preference pairs, and held-out evaluation prompts. Ouyang et al. (2022) start from labeler-written prompts and prompts submitted through an application programming interface.

Sample prompts in strata, such as everyday chat, coding, safety-sensitive requests and rare languages, with a floor per stratum so rare cases are not drowned out; the worked example shows the arithmetic.¹ Measure label quality before scaling: have two annotators label the same pairs and compute an agreement statistic that corrects for chance, such as Cohen's kappa.¹ Remove from training any prompt that also sits in an evaluation set; Lambert et al. report substantial decontamination of existing open datasets on their benchmarks.

Input: raw prompts. Output: three separated, quality-checked datasets. You now know what the later stages consume.

Step 2: Supervised fine-tuning teaches the format

Supervised fine-tuning is training the base model on demonstrations with the ordinary next-token loss, so it learns the answer format, tone and basic instruction following. Ouyang et al. (2022) fine-tune generative pre-trained transformer (GPT)-3 this way on labeler demonstrations before any feedback stage.

Input: prompts with ideal answers. Output: a model called the reference model, the starting point and the anchor for later stages. Supervised fine-tuning is stable and cheap next to the later stages, but it only imitates: it cannot tell the model that one acceptable answer is better than another. Where compute is short, adapter methods cut the cost; see low-rank adaptation (LoRA) fine-tuning. Check the result on your held-out prompts before moving on, so a later regression has a baseline to be compared with. You now have a model that answers in the right shape and needs a signal for what is better.

Step 3: Pick a training signal that actually teaches

The training signal is the score that tells the model which of its outputs to make more likely. Three main sources exist: preference pairs labeled by people, a reward model (a network trained on preference pairs to score any output), and a verifier, a program that checks the answer. Lambert et al. use DPO, introduced by Rafailov et al. (2023), for preferences and verifiable rewards for tasks with checkable answers. For the trade between the two preference routes, see DPO vs RLHF and what RLHF is.

For a coding or math model, a verifier is the natural signal, but it only teaches when some sampled answers pass and some fail. With eight samples per prompt and a 10% per-sample pass rate, 0.5695 of prompts have at least one pass and 0.4305 have none. The share that matters is prompts with mixed outcomes: one minus the all-fail share minus the all-pass share. With eight samples per prompt, 0.9825 of prompts have mixed outcomes at a 60% pass rate, but only 0.1492 at 2%. At a 90% pass rate the share falls to 0.5695 because 0.4305 of prompts are all passes.¹

A prompt where every sample passes or every one fails gives the loop nothing to learn from, so filter prompts by pass rate. Methods such as Group Relative Policy Optimization (GRPO), explained in GRPO explained, compare samples within a group, so an all-pass group has zero advantage, exactly like an all-fail group.

Step 4: Run generation, scoring and learning as a loop

The training loop alternates generation and learning. Generators sample rollouts from the current model, a scoring service (a reward-model server or a sandbox that runs tests) scores them, and a learner updates the weights. In an asynchronous design these run as separate workers joined by a queue, an orchestrator ships fresh weights to the generators, and training workers are kept apart from the fleet serving users.

Asynchrony buys throughput and costs staleness. If generating a batch takes 120 seconds and the learner updates every 40, a sample can be up to three policy versions old when it is used (illustrative timings).¹ Cap the allowed age and discard or down-weight older samples. Reinforcement-learning stages also add a Kullback–Leibler (KL) divergence penalty, a term that grows as the policy's outputs drift away from those of the reference model.

It is the main brake on reward hacking and drift, and it is why the reference copy stays in memory. Memory is the other constraint: three 8-billion-parameter copies of weights in 16-bit form, policy, reference and reward model, need 48 GB; a loop based on proximal policy optimization (PPO) adds a value model (a critic), so four copies need 64 GB, before optimizer state or activations (illustrative size).¹ The data versus model parallelism page covers how to split them across graphics processing units (GPUs).

Step 5: Evaluate and gate every release

The release gate is a fixed set of held-out checks that a candidate model must pass before it replaces the current one. Lambert et al. describe a multi-task evaluation scheme with development and unseen evaluations, and they also report training methods that did not reliably improve performance, so a gate must be able to say no.

Cover four things: the target skill (for example, tests passed on held-out coding tasks), general ability so earlier skills do not regress, safety behavior, and human preference. Ouyang et al. (2022) found that people preferred outputs of a 1.3 billion-parameter model trained with human feedback over a 175 billion-parameter GPT-3, so human comparison on your own prompt distribution can overturn what a benchmark says. Keep the unseen set untouched during development, or you tune to it. Evaluation on held-out sets gates every release.

1 · Data2 · Fine-tuning3 · Training signal4 · Training loop5 · Release gateReplay

Step-by-step description

1. Prompts, demonstrations and preference pairs.
2. Demonstrations teach the format.
3. Preferences or checks say what is better.
4. Generate, score and learn.
5. Held-out evaluation decides.

• Post-training runs in five steps: data, fine-tuning, a training signal, a training loop and a release gate.
• Evaluation on held-out sets gates every release.

Worked example

Four small calculations show what an interviewer wants to see. Every input is illustrative; the arithmetic is computed.¹

Preference-label quality. Two annotators label 200 pairs (illustrative). They agree on 150, so observed agreement is 0.75.¹ Chance agreement, from how often each picks answer A, is 0.5112, so the chance-corrected statistic is 0.4885. Raw agreement of 75% looks fine, but kappa shows the labels are far less reliable than 75% suggests. Fix it with clearer guidelines, a gold set for calibration, and majority vote: three independent annotators, each right 80% of the time, give a majority that is right 0.896 of the time (an independence assumption).

Stratified sampling. From 10,000 prompts in four strata of 7,000, 2,000, 700 and 300 (illustrative), label 1,000.¹ Proportional sampling gives 700, 200, 70 and 30. A floor of 100 per stratum plus a proportional share gives 520, 220, 142 and 118, and training weights then correct the distortion.

Text description of the diagram

• With eight samples per prompt, 0.9825 of prompts have mixed outcomes at a 60% pass rate.
• At a 90% pass rate the share falls to 0.5695 because 0.4305 of prompts are all passes.

Trade-offs and edge cases

Each signal fits a different task, and none is free.

Text description of the diagram

• Human preference pairs suit style and safety but carry noise and cost.
• A verifier suits code and math but needs checkable tasks.

Human pairs suit style and safety but carry noise and cost. A verifier suits code and math but needs checkable tasks. A reward model scores at scale but can be gamed: the policy finds outputs that score high without being good, so keep refreshing preference data against current outputs. When people are too costly, AI feedback can stand in for some labels; see Constitutional AI, and watch for the judge's own bias.

Other edge cases. A model already serving traffic needs a rollback path, and any use of logged user conversations needs consent and filtering. Reasoning-style training raises extra questions about long outputs; see how reasoning models work. Contamination between training and evaluation prompts inflates scores silently. Post-training quantization (PTQ) is a different topic: it compresses a finished model's weights into lower precision for cheaper serving, and it is not an alignment stage.

How to answer it in an interview

The spoken structure: name the five stages, say where each signal comes from, then give the one failure you would monitor.

Common mistakes

• "Post-training means RLHF." (wrong) RLHF is one stage inside it. (right) "Post-training spans data, supervised fine-tuning, a feedback stage and evaluation."
• "More fine-tuning data is always better." (wrong) Noisy examples teach noise. (right) "I measure label agreement and filter before scaling."
• "I'll evaluate on the benchmark I trained against." (wrong) The score then proves memory, not skill. (right) "Held-out, decontaminated sets gate the release."
• "Asynchronous training is just faster." (wrong) Stale samples distort the update. (right) "I cap sample age and track policy versions."
• "Any prompt works for a verifier." (wrong) Prompts always failed or always solved teach nothing. (right) "I filter prompts by pass rate."

Follow-up questions

Is RLHF considered post-training?

Yes. RLHF, as in Ouyang et al. (2022), is a feedback stage that follows supervised fine-tuning, so it is one part of post-training, not a synonym for it.

Your preference data has low annotator agreement and noisy labels. How do you measure and fix preference-data quality?

Measure chance-corrected agreement such as kappa on a shared sample, then tighten guidelines, add a gold set, and use majority vote. The worked example shows a kappa of 0.4885 behind 75% raw agreement.

How would you design a post-training pipeline for a coding model?

Use unit-test verifiers as the reward, filter prompts to those the model sometimes solves, run execution in a sandbox, and gate on held-out coding tasks plus general-ability checks. For a model that solves math problems, the design is the same with answer-matching verifiers in place of unit tests.

How would you architect an asynchronous RL post-training system for a chat LLM already serving traffic?

Separate generators, scoring service, learner and orchestrator with queues between them, cap sample staleness, and keep training workers apart from serving so user traffic is not affected.

How do you design a preference-collection pipeline with quality sampling?

Sample prompts by stratum with a floor for rare strata, double-label a slice to track agreement, and reweight in training.

Is post-training quantization (PTQ) the same thing as post-training?

No. PTQ compresses a trained model's weights for cheaper serving; post-training changes behavior through data and feedback.

Sources

1. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), 4 Mar 2022
2. Tulu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv 2411.15124 v5) — arXiv (Lambert et al., Allen Institute for AI), 14 Apr 2025
3. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023