Fine-tuning an LLM: what it is, how it works and how much memory it needs

In this article

1. Before you start
2. Why interviewers ask this
3. Fine-tuning an LLM from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Fine-tuning questions test whether you can choose the right tool and cost it, not whether you can recite a recipe. The usual probe is "how would you adapt a model to a domain", and the follow-ups are about data, forgetting and what "it worked" means. A weak answer says "train it on our data" and stops. A strong answer first asks whether prompting or retrieval would do, names the data format, says which tokens are scored and why, estimates the memory, and says how the result is checked against the base model. A common follow-up is LoRA. The fine-tuning and LLM training hub places it with the other training topics, and LoRA fine-tuning has its own page.

Fine-tuning an LLM from first principles

Instruction tuning

Instruction tuning is a kind of supervised fine-tuning that uses many different tasks, each written as a natural-language instruction with its answer, and that teaches a model to follow instructions it has not seen before.

Catastrophic forgetting

Catastrophic forgetting is the loss of abilities a model had before fine-tuning, such as reasoning or reading comprehension, that happens when the new training data pulls its weights away from the values that supported them.

In plain words

A pretrained model is like a graduate who has read a huge library but never held your job. Fine-tuning is on-the-job training with worked examples. Each says: here is a request, here is the answer we want. After enough examples the graduate answers in your house style unprompted. The analogy breaks in one place. A person remembers an old skill while learning a new one, but a model has only one set of weights, so new training can overwrite old behaviour. Stated precisely, fine-tuning runs the same next-token training as pretraining, on far less data, usually at a lower learning rate, with the loss computed on chosen tokens.

Step 1: Decide that fine-tuning is the right tool

The first step is to check that changing weights is what you need. The input is the behaviour you want; the output is a decision. Fine-tuning is good at shape: a style, a format, a task framing. The authors of Less Is More for Alignment (LIMA) argue that "almost all knowledge in large language models is learned during pretraining" and that a small instruction set mainly teaches the model which format to use (Zhou et al., 2023). If you need fresh or private facts, retrieval is usually the better first move; see fine-tuning versus prompt engineering and retrieval-augmented generation (RAG). If a better prompt reaches the target, you stop here.

Step 2: Build the example set

The second step writes the training data. The input is the target behaviour; the output is a set of examples, each a prompt and the response you would want. Quality matters more than volume. LIMA trained a 65-billion-parameter model with supervised fine-tuning on 1,000 curated prompt and response pairs with a standard supervised loss and no reinforcement learning (Zhou et al., 2023). Google's instruction-tuned model was trained on more than 60 natural language datasets written as instructions, and beat the 175-billion-parameter generative pre-trained transformer (GPT) model GPT-3 zero-shot on 20 of 25 datasets it was tested on (Wei et al., 2021). Hold some examples back; Step 5 needs them. What you now know: the data defines the behaviour.

Step 3: Choose full training or adapters

The third step sets how many weights you will change, and so whether the job fits. Full fine-tuning updates every weight. With mixed-precision Adam, each parameter costs 2 bytes for the weight, 2 for its gradient and 12 for the optimizer state, a total of 16 bytes per parameter (Rajbhandari et al., 2019, section 3.1). For a model of 7 billion parameters that is 112 GB before activations.¹ An adapter is a small set of new weights added beside a frozen layer, so only the adapter is trained.

LoRA (low-rank adaptation) is the most widely used parameter-efficient method, and it can be merged into the weights. Freezing a weight removes its gradient and optimizer state, so it costs 2 bytes instead of 16: for the 7-billion-parameter model that is 14 GB against 112 GB, an eightfold cut before activations, and the adapters add only about 67 MB.¹ QLoRA (quantized LoRA) also stores the frozen base in 4 bits and fine-tunes a 65-billion-parameter model on one 48 GB graphics processing unit (GPU) (Dettmers et al., 2023). See parameter-efficient fine-tuning methods.

Step 4: Train, and decide which tokens are scored

The fourth step is the training loop, and its one decision is which tokens are scored. The input is a prompt and a response joined into one sequence; the output is a loss. The loss is the token-level cross-entropy, the negative log-probability of each target token (Hugging Face Transformer Reinforcement Learning (TRL) supervised fine-tuning (SFT) trainer documentation, v1.16, as of October 2026; see also cross-entropy loss). For prompt and completion data the trainer scores the completion only by default, and padding is excluded with an ignore index of minus 100.

This is called loss masking: marking tokens so they add nothing to the loss. Masking the prompt is the common default, not a law: TRL scores every token for plain-text data, and one study found that keeping the loss on instructions helped when instructions were long, outputs were short or the set was small (Shi et al., 2024). Treat it as a setting to test, and keep it fixed when you compare runs, because a loss over all tokens and a loss over responses measure different things.

Step 5: Check the result and what it forgot

The fifth step compares the tuned model with the base model. The input is the held-out examples from Step 2 plus a few general tasks; the output is a decision to ship, change the data or stop. Run both models on the same held-out prompts and read the outputs, then add a task score if one exists; see LLM evaluation. Also test something the tuning was not about.

A continual fine-tuning study of models with 1 to 7 billion parameters found catastrophic forgetting in domain knowledge, reasoning and reading comprehension, and found that forgetting got worse as model size increased in that range (Luo et al., 2023). Common mitigations are a gentler learning rate for full training (adapters typically use a higher one, around 1e-4, in TRL's guidance), adapters, which one study found forget less (Biderman et al., 2024), and replay. Replay is the practice of mixing some general data into the training set.

1 · Pick the tool2 · Build examples3 · Choose the method4 · Train5 · CheckReplay

Step-by-step description

1. Decide fine-tuning beats prompting.
2. Write prompt and response pairs.
3. Full training or adapters.
4. Loss on chosen tokens.
5. Compare with the base model.

• Fine-tuning has five steps: decide, build examples, choose the method, train on responses and check.
• The last step compares the tuned model with the base model on held-out examples.

Worked example

Take a model of 7 billion parameters, with a hidden size of 4,096 and 32 layers (an illustrative shape close to common 7B designs). Full fine-tuning needs 16 bytes per parameter, or 112 GB. LoRA keeps the base weights frozen in half precision, 2 bytes each, which is 14 GB, and trains adapters on the query and value matrices of every layer at rank 8.

Each adapted matrix adds 8 times (4,096 plus 4,096) weights, so the adapters hold 4,194,304 weights, about 0.06% of the model. Trained in full precision with Adam, they take 67 MB, so the total is about 14.07 GB. QLoRA stores the base in 4 bits, 0.5 bytes each, about 3.5 GB, for a total near 3.57 GB.¹ The LoRA total is about 14.07 GB and the QLoRA total about 3.57 GB, excluding activations, which depend on batch size and sequence length.

Text description of the diagram

• Full training of a 7-billion-parameter model needs about 112 GB for weights, gradients and optimizer state.
• LoRA totals about 14.07 GB and QLoRA about 3.57 GB, excluding activations.

Two smaller numbers show the training mechanics. With 1,000 examples, a batch of 16 and 3 passes over the data, the optimizer takes 63 steps per pass and 189 in total. And if a sequence has 40 prompt tokens with an average loss of 3.0 and 60 response tokens with an average loss of 1.2 (illustrative values), the loss over all tokens is 1.92, while the loss over the response alone is 1.2; the first number is higher only because the prompt tokens in this illustration are harder to predict. With templated prompts the whole-sequence loss can instead be lower, which is why the two numbers must never be compared across runs.¹

Trade-offs and edge cases

Data quality over quantity. A small, clean set can beat a large noisy one: LIMA's 1,000 pairs were enough to produce a model that human raters judged equivalent to or better than GPT-4 in 43% of comparisons (Zhou et al., 2023). The flip side is that errors in a small set are learned just as faithfully, so review the data before training.

Facts versus behaviour. Fine-tuning shapes how a model answers more than what it knows, so a model tuned on last quarter's documents will still invent this quarter's facts; use retrieval for that. Forgetting versus adapters. Adapters leave the base weights untouched, which limits the damage, but the model's behaviour still changes, so the general-ability check in Step 5 still applies.

Where SFT stops. Supervised fine-tuning teaches imitation of the examples. When you want the model to prefer one good answer over another, a preference stage follows; InstructGPT used reinforcement learning from human feedback (RLHF) after supervised tuning, and its 1.3-billion-parameter model was preferred by human raters to the 175-billion-parameter GPT-3 (Ouyang et al., 2022). See direct preference optimization (DPO) versus RLHF. Memory beyond model states. The 16-byte figure covers model states only: 2 bytes of weight, 2 of gradient and 12 of optimizer state, which is a 32-bit master copy plus Adam's two moments.

Activations come on top, and LoRA does not remove them, because gradients still flow back through the frozen layers to reach every adapter. The 12 bytes of optimizer state are covered in optimizer states, and 4-bit storage in LLM quantization. The LoRA authors measured a smaller cut on GPT-3 175B, from 1.2 TB to 350 GB, about 3 times, with 10,000 times fewer trainable parameters (Hu et al., 2021, section 4.2): their full-training figure works out to about 7 bytes per parameter, not 16, and 350 GB is the 2-byte frozen base alone.¹

How to answer it in an interview

The spoken structure is: tool choice, data, method and memory, then the check.

Common mistakes

Comparing losses across masking settings. (wrong) "Run B's loss is 1.2 and run A's is 1.92, so B learned more." (right) "Run A scored prompt and response, run B only the response, so the numbers cover different tokens; I compare both on the same held-out responses." The worked example shows how the same model gives both numbers.¹

Fine-tuning for facts. (wrong) "I will train it on the new policy documents so it knows them." (right) "I will retrieve the documents at question time and tune only for format."

No held-out check. (wrong) "The training loss went down, so it worked." (right) "I compared base and tuned outputs on held-out prompts."

Ignoring forgetting. (wrong) "It does the new task, so we are done." (right) "I also tested general tasks, because forgetting was observed across 1 to 7 billion parameter models."

Text description of the diagram

• Runs that score different tokens report losses that cannot be compared.
• Tuning to add fresh facts is better handled by retrieval.
• A held-out check and a general-ability check catch forgetting.

Follow-up questions

What is instruction tuning?

Instruction tuning is supervised fine-tuning on many tasks written as instructions, so the model learns to follow instructions it has not seen. The instruction-tuning study tuned a 137-billion-parameter model on more than 60 datasets and found that the number of datasets, model scale and natural-language instructions were key to the gain (Wei et al., 2021).

How much data do you need to fine-tune an LLM?

It depends on the goal, and curation matters more than count. LIMA used 1,000 curated pairs for a general assistant style (Zhou et al., 2023); a narrow format may need fewer, and a new skill more. Start small, measure on the held-out set, and add data where the errors cluster.

How do you avoid catastrophic forgetting?

Use a lower learning rate, train adapters instead of all weights, and mix some general examples into the training set. A study of continual instruction tuning reported that general instruction tuning first helped reduce forgetting in later tuning (Luo et al., 2023). Then measure it.

How do you fine-tune an LLM?

First confirm that prompting or retrieval cannot reach the goal, then write a few hundred to a few thousand prompt and response pairs and hold some back. Pick full training or LoRA by memory, train (usually scoring only the responses), and compare the tuned and base models on the held-out set and a few general tasks; Steps 1 to 5 above walk through each.

What if the model has 70 billion parameters?

Full fine-tuning then needs 16 × 70 billion = 1,120 GB of model states, so it must be sharded across many GPUs. LoRA on a frozen 16-bit base needs about 140 GB, and QLoRA's 4-bit base about 35 GB, which is why QLoRA fits a 65-billion-parameter model on one 48 GB GPU (Dettmers et al., 2023).¹

Retrieval augmented generation or fine-tuning?

Retrieve when the answer depends on facts that change or are private; fine-tune when the model must change style, format or task behaviour. Many systems do both, with the tuned model reading retrieved text; fine-tuning versus prompt engineering compares the cheaper options first.

Sources

1. Training language models to follow instructions with human feedback — arXiv (Ouyang et al.), 4 Mar 2022
2. Finetuned Language Models Are Zero-Shot Learners — arXiv (Wei et al.), 3 Sep 2021
3. LIMA: Less Is More for Alignment — arXiv (Zhou et al.), 18 May 2023
4. LoRA: Low-Rank Adaptation of Large Language Models — arXiv (Hu et al.), 17 Jun 2021
5. QLoRA: Efficient Finetuning of Quantized LLMs — arXiv (Dettmers et al.), 23 May 2023
6. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — arXiv (Rajbhandari et al.), 4 Oct 2019
7. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning — arXiv (Luo et al.), 17 Aug 2023
8. LoRA Learns Less and Forgets Less — arXiv (Biderman et al.), 15 May 2024
9. Instruction Tuning With Loss Over Instructions — arXiv (Shi et al.), 23 May 2024
10. SFT Trainer (Hugging Face TRL documentation) — Hugging Face (date = read date), 24 Oct 2026