In this article
Before you start
This page is one step of the how large language models are trained path.
Why interviewers ask this
The question "how will you finetune an LLM?" is open on purpose, and "I would train it on our data" is the answer that ends the discussion early. The interviewer wants to hear a decision process: whether to tune at all, which method fits the hardware, what the data looks like and how success is measured.
A weak answer names one tool and one command. A strong answer separates the goal (behaviour, format, preference) from the method (full tuning, LoRA, preference tuning) and says how the result is evaluated. A Staff-level answer also names the cost of each choice, such as memory, forgetting and serving, and says when prompting or retrieval is the better tool. This question is reported by candidates in design and machine learning (ML) rounds; no public source ties it to a specific company.
Finetuning from first principles
- Full fine-tuning
- Full fine-tuning is a training method that updates every parameter of a pretrained model on new examples, which can fit the task closely but needs far more memory and storage than methods that train only a small added set of weights.
In plain words
Fine-tuning is continued training of an already trained model on a smaller, more specific set of examples. Think of a hired chef who already cooks well. You do not send the chef back to cooking school; you hand over your menu and let them practise those dishes. The analogy breaks in one place. Practising your menu can make the chef slightly worse at dishes outside it, and models show the same drift.
In one precise sentence, fine-tuning updates a pretrained model's parameters, or a small added set of them, to lower the loss on your examples.
Step 1: Decide whether to tune at all
The first step is the goal. Behaviour, tone and output format are what tuning changes well. Facts that change often are safer to supply at request time through the prompt or retrieval. Fine-tuning versus prompt engineering covers that choice in depth.
Input: the failure you see with the base model. Output: a decision, either a better prompt, retrieval, or a finetune with a stated goal such as "always answer in this JavaScript Object Notation (JSON) format". You now know what the tune is for, which fixes what data and what test you need.
Step 2: LoRA trains two small matrices beside each frozen weight
Full fine-tuning retrains every parameter. Hu et al. (2021) note that this becomes less feasible as models grow, and that keeping a separate 175-billion-parameter copy for each task is prohibitively expensive. LoRA is a method that freezes the pretrained weights and learns the change to a weight matrix as the product of two small matrices of rank r.
Take one weight matrix with 4,096 rows and 4,096 columns (illustrative). Full tuning updates 16,777,216 numbers in it. LoRA at rank 8 trains 65,536, about 0.39% of them.1 The count is r times the sum of the two sides, against rows times columns. Hu et al. adapted the query and value projections of attention, so an illustrative 32-layer model that adapts two matrices per layer holds 4,194,304 trainable parameters.1 Why so few are enough: Hu et al. hypothesise that the change in weights during adaptation has a low "intrinsic rank", so a rank-r product can capture most of it (Hu et al., 2021, §1). Once merged, the adapter adds no inference latency; the LoRA page goes further.
Step 3: Train on prompt-and-answer examples
Training runs on examples that show the behaviour you want: a prompt and the answer a good model would give. Quality matters more than volume, so clean, consistent examples beat a large noisy pile. Split off two sets you never train on: a validation set to choose when to stop, and a test set used once at the end.
Input: the examples, formatted the way the model expects its prompts. Output: an adapter whose loss on the validation set stops improving. Keep the learning rate modest and stop when validation loss turns upward, because a model that keeps training will memorise the examples. If memory is the constraint, the quantised variant, QLoRA (quantised LoRA), keeps the frozen base in 4-bit form, as Dettmers et al. (2023) describe.
Step 4: Evaluate, then merge or add preference tuning
Evaluate once on the test set and on a few general tasks, to catch forgetting. For a preference goal such as style, add preference tuning after supervised fine-tuning (SFT): RLHF (Ouyang et al., 2022) or DPO (Rafailov et al., 2023), which trains on preferences directly. See DPO versus RLHF and what RLHF is.
Input: the trained adapter. Output: a go or no-go decision backed by test-set numbers. If the result meets the goal, merge the adapter into the base weights so serving is unchanged (for a QLoRA adapter, dequantise the base first), or keep adapters unmerged if one base must serve many tasks.
Step-by-step description
- Decide whether tuning is needed.
- Freeze the base, add small matrices.
- Learn from prompt-answer examples.
- Test on held-out data, then merge.
- Fine-tuning starts from a pretrained model, so the first decision is whether tuning is needed at all.
- LoRA freezes the pretrained weights and trains two small matrices per adapted layer.
- Training runs on prompt and answer examples that show the behaviour you want.
- A held-out test set decides whether the tune worked, and the adapter is merged for serving.
In short, fine-tuning starts from a pretrained model, so the first decision is whether tuning is needed at all; LoRA freezes the pretrained weights and trains two small matrices per adapted layer; training runs on prompt and answer examples that show the behaviour you want; and a held-out test set decides whether the tune worked, after which the adapter is merged for serving.
The memory gain is the usual follow-up. Charge an illustrative 16 bytes per trained parameter (weights, gradients and optimiser state, an assumption, not a measurement) and only 2 bytes per frozen weight, and ignore activations. Illustrative memory for a 7-billion-parameter model is 112 GB with full tuning and 14.07 GB with a LoRA adapter.1 Measured gains are smaller because activations and other overheads remain: Hu et al. (2021) report about three times less graphics processing unit (GPU) memory than full fine-tuning of generative pre-trained transformer (GPT)-3 175B with Adam. See the memory guide to fine-tuning an LLM for real numbers.
Text description of the diagram
- Illustrative memory for a 7-billion-parameter model is 112 GB with full tuning and 14.07 GB with a LoRA adapter.
- LoRA freezes the pretrained weights and trains two small matrices per adapted layer.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How will you finetune LLMs?
First I check whether tuning is the right tool, because prompting or retrieval fixes many problems more cheaply. If it is, I start from an instruction-tuned base and build clean prompt-and-answer examples, keeping a validation set and a separate test set I never train on. I train a LoRA adapter, meaning small low-rank matrices beside frozen weights, which needs far less memory than full tuning and merges back with no extra inference latency. If memory is tight I quantise the frozen base, as QLoRA does. I judge it on held-out tasks and a few general benchmarks to catch forgetting, not on training loss. For a style preference I add DPO after supervised tuning.
What makes it Staff-level: it starts with the decision to tune, names the cost and the failure (memory, forgetting) and ends with how success is measured.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "Train it on our data." | Continue training on prompt-answer examples with a method that fits the hardware. | Chooses between prompting, retrieval, LoRA and full tuning by the failure observed. |
| Depth | Names one library. | Explains that LoRA freezes weights and trains small matrices. | Counts trainable parameters and memory, and explains merging. |
| Trade-offs | None. | Mentions memory against quality. | Names forgetting, data quality and rank as dials, with tests for each. |
| Evaluation | Watches training loss. | Uses a held-out set. | Adds general benchmarks and task metrics before any merge. |
| Communication | Lists tools. | Orders the steps. | Gives a decision first, then the method, then the check. |
Three variants of this question
How would you finetune an LLM on one GPU?
Use LoRA, and quantise the frozen base if needed (QLoRA), so that only the small adapter needs gradients and optimiser state. Train with a small batch and accumulate gradients if memory is still tight.
How would you finetune an LLM to follow your company's tone?
Collect examples of good answers in that tone and run supervised tuning with an adapter. Judge by held-out prompts rated against a written style guide, since tone is a behaviour tuning handles well.
How would you finetune an LLM on a new document set?
Treat tuning as shaping behaviour, and supply changing facts through retrieval at request time, so they can be updated without retraining. Tune only if the format or reasoning over the documents also needs to change.
Common mistakes
- "I would fine-tune because the model lacks our facts." (wrong) That treats tuning as a database. (right) "I would supply changing facts at request time and tune for behaviour."
- "LoRA is always worse than full tuning." (wrong) The LoRA authors report it on par or better on the models they tested. (right) "LoRA is the default, and I move to full tuning only if held-out results show a gap."
- "LoRA slows inference." (wrong) After merging, the weights are an ordinary matrix. (right) "I merge the adapter, so serving cost is unchanged."
- "I stop when training loss is low." (wrong) That rewards memorising. (right) "I stop on validation loss and check general tasks on the test set."
- "DPO is reinforcement learning with a reward model." (wrong) DPO trains on preferences directly. (right) "DPO needs no separate reward model."
Follow-up questions
How do you choose the LoRA rank?
Start small, since the rank sets how many parameters the update has, and raise it only if validation results plateau below the goal. A larger rank costs proportionally more trainable parameters, as the count in Step 2 shows.
How do you keep the model from forgetting what it knew?
Freeze the base by using an adapter, keep the learning rate modest, mix in some general examples, and re-run general benchmarks before merging. The alignment tax page covers the same trade-off.
How do you know the finetune worked?
Compare against the untuned base on a held-out set that matches real use, with the metric tied to the goal from Step 1. A lower training loss alone proves nothing.
When would you use DPO instead of more supervised tuning?
Use DPO when you have pairs of better and worse answers and the goal is a preference, such as style or safety, rather than a new format. Supervised examples show what to say; preference pairs show which of two answers is better.
- Decide whether to tune before choosing a method; behaviour is tuned, changing facts are supplied.
- LoRA freezes the base and trains two small matrices, about 0.39% of one illustrative matrix at rank 8.
- Judge a tune on held-out data and general tasks, never on training loss.
- Merge the adapter for serving; add preference tuning only for preference goals.
Sources
- LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685) — arXiv (Hu et al.), 17 Jun 2021
- QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314) — arXiv (Dettmers et al.), 23 May 2023
- Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), 4 Mar 2022
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023
