What is parameter-efficient fine-tuning (PEFT), and how does LoRA work?

In this article

1. Before you start
2. Why interviewers ask this
3. Parameter-efficient fine-tuning from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the large language model (LLM) training and fine-tuning learning path and builds on the transformer architecture.

Why interviewers ask this

Fine-tuning a large model is expensive, so questions on how to do it cheaply are reported by candidates in machine learning and LLM interviews. One reported form asks how PEFT techniques such as LoRA or QLoRA work mathematically and when to choose them over full fine-tuning.

The consequence of a vague answer is a quick follow-up on memory or rank, where the candidate has no numbers. A weak answer says "LoRA trains fewer parameters." A strong answer says what is frozen, what is added, how the added part is shaped, and why the memory saving is smaller than the parameter saving. A Staff-level answer adds the serving side (merge the adapter or keep one base with many adapters) and a rule for when full fine-tuning still wins.

Parameter-efficient fine-tuning from first principles

Parameter-efficient fine-tuning

Parameter-efficient fine-tuning is a family of methods that adapt a pretrained model to a new task by training only a small number of added or selected parameters while the base weights stay frozen, so each task stores a small add-on instead of a full copy of the model.

Adapter

An adapter is a small trainable module or set of matrices attached to a frozen base model that holds everything the model learns for one task, so the same base can serve many tasks by swapping adapters.

In plain words

Think of a large textbook that you want to adapt for one class. Rewriting every page is slow and needs a new printing. Instead you slip a few sticky notes into the margins. The book stays the same. The notes carry the change. The analogy breaks in one place. A sticky note sits beside a page, while a LoRA update is added into the page's numbers and can be merged so that no notes remain.

In one precise sentence, PEFT freezes the pretrained weights and trains a small extra set of parameters so that the model plus the add-on does the new task. The Hugging Face PEFT documentation describes it as a framework for adding trainable parameters at chosen places in a base model. The PEFT methods guide lists the families.

Step 1: Full fine-tuning updates every weight and stores a full copy

Full fine-tuning is the usual way to adapt a model: every weight gets a gradient and is updated. The Hugging Face PEFT documentation calls this exceedingly costly and impractical because of the number of parameters in models today.

The cost has two parts. Training needs memory for weights, gradients and optimizer state, which is the extra numbers the Adam optimizer keeps for each trainable weight. Storage needs a whole new model per task: Houlsby et al. (2019) note that an entire new model is required for every task. For a 175-billion-parameter model, Hu et al. (2021) call an independent full copy per task prohibitively expensive. You now know the problem PEFT solves: training memory and per-task storage both scale with the whole model.

Step 2: Freezing the base removes optimizer memory, not weight memory

Freezing the base model means its weights receive no gradient and no optimizer state, so only the small add-on needs them. The Hugging Face Transformer Reinforcement Learning (TRL) documentation states that PEFT keeps the base model frozen while training a small number of additional parameters.

The weights still sit in memory, because the forward pass needs them. In an illustrative accounting for a 7-billion-parameter model with 2 bytes per weight, 2 bytes per gradient and 8 bytes of Adam state, full tuning needs about 84 GB before activations.¹ With a frozen base and a LoRA add-on, the same accounting gives about 14.05 GB, which is far less but still more than the weights alone.¹ Parameters fall by a huge factor; memory falls by a smaller one. The next step shows the add-on itself.

Step 3: LoRA trains two thin matrices per adapted weight matrix

LoRA is a PEFT method that freezes the pretrained weights and injects trainable rank decomposition matrices into each Transformer layer. For one weight matrix W0 of shape d by k, LoRA does not change W0. It trains two thin matrices, B of shape d by r and A of shape r by k, and the layer uses $W=W_0+B·A$.

The number r is the rank, and it is much smaller than d or k. Hu et al. (2021, §4.1) scale the update by α/r, where α is a constant, so the layer computes $h=W_{0}x+\frac{\alpha }{r}BAx$. They initialise A with random Gaussian values and B with zeros, so the product of B and A is zero at the start and training begins exactly from the pretrained model.

The bet is that the weight changes needed for a task have a low intrinsic rank, which is the hypothesis stated in the first version of the LoRA paper. The PEFT LoRA guide says LoRA is typically applied to attention blocks only. For a 4,096 by 4,096 matrix, one full update has 16,777,216 numbers while rank 8 trains 65,536.¹ The LoRA fine-tuning guide goes deeper.

Step 4: QLoRA stores the frozen base in 4-bit form

QLoRA is a method that trains LoRA adapters on top of a frozen base model stored in 4-bit form. Quantization is the storing of weights with fewer bits each, at some loss of precision. Dettmers et al. (2023) backpropagate gradients through the frozen 4-bit quantized model into the LoRA adapters, so only the adapters are updated. QLoRA stores the weights in 4-bit NormalFloat (NF4), a data type the abstract calls information theoretically optimal for normally distributed weights, and the weights are dequantized to 16-bit BrainFloat (BF16) for each matrix multiplication, so the computation runs in BF16.

This attacks the part Step 2 left alone, the weights. The QLoRA paper reports that the average memory for tuning a 65-billion-parameter model falls from over 780 GB to under 48 GB of GPU (graphics processing unit) memory, without worse predictive performance than a 16-bit fully tuned baseline on their setups. The same abstract names two more tricks, double quantization of the quantization constants and paged optimizers for memory spikes. In the illustrative 7-billion-parameter accounting, a 4-bit base is 3.5 GB of weights and the total is about 3.55 GB.¹

Step 5: The adapter merges into the base, or many adapters share one base

A merged LoRA adapter adds no inference latency, because adapter weights can be merged with the base model: the product of the two thin matrices has the same shape as W0, so the sum can be stored once and the layer runs as before. Unmerged adapters add a small extra matrix multiplication per adapted layer, and merging into a 4-bit base requires dequantizing the weights first. Hu et al. (2021) contrast this with inserted adapters, which add inference latency.

You can also not merge. Keep one frozen base and store one small adapter per task, then swap them. That is the storage idea of Houlsby et al. (2019): the original network stays fixed, giving a high degree of parameter sharing, and new tasks are added without revisiting old ones. Merging favours speed and simplicity; swapping favours many tasks on one base.

1 · Full tuning2 · Freeze the base3 · Add two thin matrices4 · Quantize the base5 · Merge and serveReplay

Step-by-step description

1. Every weight is updated and each task needs a full copy.
2. No gradients or optimizer states, but the weights stay in memory.
3. LoRA trains B and A of rank r per weight matrix.
4. QLoRA holds the frozen base in 4-bit form.
5. The adapter folds into the base, so no latency is added.

• Full fine-tuning updates every weight and stores a full copy per task.
• Freezing the base removes gradients and optimizer states but keeps the weights in memory.
• LoRA trains two thin matrices per weight matrix.
• QLoRA stores the frozen base in 4-bit form.
• A merged adapter adds no inference latency.

Worked example

Take one square weight matrix with 4,096 rows and columns, as an illustrative size. A full update trains 16,777,216 numbers. LoRA at rank 8 trains rank times the sum of the two sides, 8 times 8,192, which is 65,536 numbers, about 0.39% of the full count and 256 times fewer.¹ Doubling the rank doubles the count: 131,072 at rank 16 and 524,288 at rank 64.¹

Now scale to a whole model, again illustrative: 32 layers, two attention matrices of that size per layer, rank 8. That is 4,194,304 trainable numbers, about 0.06% of a 7-billion-parameter model.¹ For real models, Hu et al. (2021) report up to 10,000 times fewer trainable parameters and 3 times less GPU memory than full tuning of a 175-billion-parameter model with Adam.

The memory rows count weights, gradients and Adam state only, with 2, 2 and 8 bytes per parameter; activations and real framework overheads are left out.¹ A common mixed-precision setup also keeps a 32-bit master copy of the weights, so real full-tuning memory is higher than this illustrative 84 GB.

Text description of the diagram

• Illustrative training memory for a 7-billion-parameter model is 84 GB with full tuning.
• It is 14.05 GB with LoRA and 3.55 GB with QLoRA, before activations.

Trade-offs and edge cases

PEFT trades some flexibility for cost. Hu et al. (2021) report that LoRA performs on par or better than full tuning on the models they tested, and the PEFT guide says quality is comparable. Earlier efficient methods sometimes failed to match full tuning, which the first version of the LoRA paper describes as a trade-off between efficiency and quality. Parity is reported on those benchmarks, not proven for every task.

Text description of the diagram

• Full fine-tuning trains every weight and stores a full copy per task.
• Inserted adapters add inference latency.
• LoRA can be merged into the base, so it adds no inference latency.

Edge cases follow from the mechanism. A low rank may not capture a large change: Biderman et al. (2024) report that LoRA underperforms full fine-tuning on code and math continued pretraining and instruction tuning, while forgetting less of the source domain. So a task far from the pretraining data may need a higher rank or full tuning. Adapting more modules, for example all linear layers rather than attention only, raises the parameter count and can improve quality.

An adapter trained on one base model does not transfer to another. For serving many tasks, keep one frozen base and several unmerged adapters, accepting the small extra matrix multiplication per layer. Houlsby et al. (2019) report that their adapters came within 0.4% of full tuning on a language-understanding benchmark while adding 3.6% parameters per task. A smaller route to cheap models is a small language model or knowledge distillation.

How to answer it in an interview

The spoken structure: say what is frozen, what is added, why memory falls less than parameters, then give the decision rule and its limit.

Common mistakes

• "Freezing the model makes it small." (wrong) The frozen weights stay in memory. (right) "Freezing removes gradients and optimizer states; QLoRA shrinks the weights."
• "QLoRA is a different method from LoRA." (wrong) It is LoRA on a 4-bit base. (right) "QLoRA trains LoRA adapters through a frozen quantized model."
• "LoRA slows inference." (wrong) A merged adapter adds nothing; only unmerged adapters add a small extra matrix multiplication. (right) "After merging, the layer is the same size as before; inserted adapters are the ones that add latency."
• "PEFT matches full tuning on everything." (wrong) Parity is reported on specific benchmarks. (right) "It is comparable in the reported results; I test it on my task."

Follow-up questions

How do PEFT techniques like LoRA or QLoRA work mathematically?

LoRA keeps W0 fixed and learns the product of B and A, with B of shape d by r and A of shape r by k, so the trainable count is r times d plus k instead of d times k. The layer computes W0 x plus alpha over r times B A x; A starts random Gaussian and B starts at zero, so training begins from the pretrained model. QLoRA stores W0 in 4-bit NormalFloat (NF4), dequantizes it to BF16 for each matrix multiplication, and sends gradients through it to the adapters only.

When would you choose PEFT over full parameter fine-tuning?

Choose PEFT when you have many tasks, a limited GPU budget, or want one shared base with small per-task adapters. Choose full tuning when the task differs a lot from pretraining and you have the data and compute; this rule is editorial judgement. The fine-tuning guide covers memory in more depth, and fine-tuning vs prompt engineering covers the cheaper alternative.

Does LoRA slow down inference?

Not if you merge it: the PEFT documentation says adapter weights can be merged with the base model, so no latency is added. Unmerged adapters add a small extra matrix multiplication per layer.

What is the difference between LoRA and QLoRA?

LoRA trains low-rank adapters on a frozen base. QLoRA does the same but holds the base in 4-bit form, which is what lets a 65-billion-parameter model fit on one 48 GB GPU in the paper's report.

What happens if you raise the rank?

Trainable parameters grow linearly with r: in the worked example, rank 8 gives 65,536 numbers and rank 64 gives 524,288 for one matrix. A higher rank can fit a larger change but costs more memory and storage.

Sources

1. LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685) — arXiv (Hu et al., ICLR 2022), 17 Jun 2021
2. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314) — arXiv (Dettmers et al., NeurIPS 2023), 23 May 2023
3. Parameter-Efficient Transfer Learning for NLP (arXiv 1902.00751) — arXiv (Houlsby et al., ICML 2019), 2 Feb 2019
4. PEFT quicktour — Hugging Face PEFT documentation (living page, read 2026-10-26), 26 Oct 2026
5. LoRA conceptual guide — Hugging Face PEFT documentation (living page, read 2026-10-26), 26 Oct 2026
6. PEFT integration — Hugging Face TRL documentation (living page, read 2026-10-26), 26 Oct 2026
7. LoRA Learns Less and Forgets Less (arXiv 2405.09673) — arXiv (Biderman et al., 2024), 15 May 2024