What are PEFT methods?

In this article

1. Before you start
2. Why interviewers ask this
3. PEFT methods from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the fine-tuning and large language model (LLM) training guide, which places adaptation after pretraining.

Why interviewers ask this

Full fine-tuning of an LLM needs memory for gradients and optimizer states on every weight, about eight times the memory of the weights alone, and PEFT is the standard answer. Interviewers use it to separate candidates who name LoRA from candidates who can say what is frozen, what is trained, what it saves and what it costs at inference. It comes up as "how would you adapt a 7 billion parameter model on one graphics processing unit (GPU)", "LoRA versus adapters", and "does LoRA slow serving".

A weak answer lists acronyms. A strong answer picks a method for a stated constraint, counts the trainable parameters, and knows which methods merge into the weights and which leave extra work at inference.

PEFT methods from first principles

In plain words

PEFT

PEFT, short for parameter-efficient fine-tuning, is a family of adaptation methods that train only a small number of added or selected parameters while the rest of the pretrained weights stay unchanged. The result is one small file per task instead of one full model copy.

Frozen weights

Frozen weights are parameters that are loaded and used in the forward pass but excluded from gradient updates, so they need no gradients and no optimizer states.

Soft prompt

A soft prompt is a set of learned embedding vectors, not words, that is added to a model's input or to its attention keys and values, and is trained by gradient descent while the model stays frozen.

Think of a pretrained model as a printed textbook. Full fine-tuning rewrites the pages. PEFT leaves the pages alone and adds sticky notes: a few notes at the front, a few in the margins of each chapter, or a few corrections pinned to particular lines. Each note set is cheap to store and you can swap sets for different courses. The analogy breaks in one place. Real sticky notes sit outside the text, but some PEFT methods, like LoRA, can be merged so the corrections become part of the page.

In one precise sentence, PEFT trains a few new parameters, chosen by method, and reuses the frozen model for everything else.

Step 1: Freeze the base model

Freezing means the pretrained weights get no gradient and no optimizer state, and that is where the memory saving comes from. The PEFT library documentation describes the methods as fine-tuning only a small number of extra parameters, with performance comparable to full fine-tuning, which is its stated goal; it is a claim of the library, not a result for every task.

A frozen weight still occupies memory, because the forward pass needs it, but it costs two bytes in fp16 (16-bit floating point) and nothing for gradients or the two Adam states described on the optimizer states page. The backward pass still runs through the frozen layers to reach the trainable piece, so activation memory and most of the compute remain.

The input is a pretrained model. The output is the same model with its gradients switched off, ready to receive a small trainable piece.

Step 2: Add a small trainable piece where the method puts it

The methods differ in where the new parameters go. Prompt tuning places them at the input: Lester et al. (2021) learn soft prompts, and for a Text-to-Text Transfer Transformer (T5) model with 11 billion parameters a five-token prompt needs only 20,480 parameters per task. Prefix tuning places them at every layer: Li and Liang (2021) learn a prefix and report training only 0.1 percent of the parameters. Adapters place small bottleneck layers inside each block: Houlsby et al. (2019) report coming within 0.4 percent of full fine-tuning on the General Language Understanding Evaluation (GLUE) benchmark while adding 3.6 percent parameters per task for Bidirectional Encoder Representations from Transformers (BERT).

A fourth method, (IA)³ (infused adapter by inhibiting and amplifying inner activations) from Liu et al. (2022), scales the keys, values and feed-forward activations by learned vectors, and a fifth, LoRA, learns an update to the weights themselves.

The input is the frozen model and a choice of location. The output is the same model with extra trainable parameters attached.

Step 3: LoRA adds a low-rank update that can be merged

LoRA, short for low-rank adaptation, keeps the frozen weight matrix $W_0$ and learns an update as the product of two thin matrices, so the layer computes $h=W_{0}x+(B·A)x$. Hu et al. (2021) set $A$ to random Gaussian values and $B$ to zero, so the update starts at zero, and scale it by alpha over the rank $r$. The rank is their shared inner width and caps how complex the update can be.

The payoff is a merge. After training you add the scaled $B·A$ to $W_0$ once, so LoRA adds no inference latency by construction. Scaling vectors fold in too, since a learned scale times $Wx$ equals the scaled $W$ times $x$ (Liu et al.). Adapters cannot fold, because they add layers that run in sequence; the LoRA paper measured slow-downs of over 30 percent at batch size 1.

For a 4,096 by 4,096 matrix, a full update has 16,777,216 entries and a rank-8 LoRA update has 8 times (4,096 plus 4,096), which is 65,536, so 256 times fewer¹.

Step 4: Train and store only the small piece

Only the new parameters receive gradients and optimizer states, so training memory and the saved file both shrink. The LoRA paper reports cutting training memory for the generative pre-trained transformer (GPT) model GPT-3 175B from 1.2 TB to 350 GB, and the checkpoint from 350 GB to 35 MB, roughly 10,000 times, with rank 4 on the query and value matrices. One frozen copy then serves many tasks, because you swap the small file instead of the model, and the same file is easy to version, test and roll back.

The input is a task dataset. The output is a file of a few megabytes that, loaded onto the frozen model, specialises it.

1 · Freeze2 · Add a piece3 · Fold in4 · Train and storeReplay

Step-by-step description

1. The pretrained weights stop changing.
2. A prompt, prefix, adapter or scale joins the model.
3. A LoRA update B times A can merge into the weights.
4. Gradients and states only for the piece; one file per task.

• The pretrained weights stay frozen.
• Only the small added piece is trained and stored for each task.

Worked example

The count of trainable parameters follows from the model shape. Take a model shaped like the 7 billion parameter model in Table 2 of the LLaMA paper: a width of 4,096, 32 layers and 6.7 billion parameters in total. The LoRA paper gives the count as $|\mathrm{\Theta }|=2\times \hat{L}\times d_{model}\times r$, where $\hat{L}$ is the number of weight matrices that get an update. With rank 8 on the query and value matrices, $\hat{L}$ is 64 and the count is 4,194,304¹.

The memory column is our accounting, built on the ZeRO paper rule from the optimizer states page: frozen weights cost two bytes each and trainable weights cost 16 bytes each for the fp16 copy, gradient and mixed-precision Adam states, with activations excluded¹. The prefix and prompt counts assume direct learned vectors, with no reparameterisation network, and the adapter count assumes two adapters per layer. The inputs are illustrative choices, not recommendations.

Text description of the diagram

• LoRA at rank 8 on query and value trains 4,194,304 parameters.
• Prompt tuning with 20 virtual tokens trains 81,920 parameters.

The largest saving is not the trainable count but the optimizer states that disappear: 13.4 GB of frozen fp16 weights remain, against 107.2 GB when every weight is trained¹. Quantizing the frozen weights is the next lever. The QLoRA paper reports finetuning a 65 billion parameter model on a single 48 GB GPU by combining 4-bit NormalFloat weights, double quantization that saves about 0.37 bits per parameter, and paged optimizers. At four bits the 6.7 billion frozen weights of our example would occupy about 3.4 GB before the quantization constants¹. The LLM quantization page explains the formats.

Trade-offs and edge cases

Pick the method by what you must preserve. For a fixed serving stack, choose a merge-able method; for many tasks on one deployed model, choose a small swappable file.

Text description of the diagram

• Adapters add layers that run in sequence.
• Scaling vectors and LoRA fold into the weights, so they add no inference work once merged.

Prompt and prefix methods consume context. Soft prompts and prefixes are extra positions the model attends to, so they lengthen the attention context and can interact with the key-value (KV) cache. Lester et al. report that prompt tuning becomes more competitive as the model grows, closing the gap with full tuning above billions of parameters, so it is weakest for small models. Adapters train the most parameters of the cheap methods in the worked example, 33.8 million, and add sequential layers that cost latency.

Quality is not always equal to full fine-tuning. The adapter paper reports a 0.4 percent gap on GLUE, and the prefix-tuning paper reports it matches full fine-tuning in some settings while underperforming in others. Reported practice differs by task and scale, so measure on your own held-out set. The fine-tuning versus prompt engineering page covers when to adapt weights at all.

LoRA has its own limits. Rank is a capacity cap: setting rank to the rank of the pretrained matrix roughly recovers full fine-tuning in the paper's argument. The paper found rank 1 already competitive on query and value for its tasks, but warns that a small rank may not suffice for a task far from pretraining, such as a new language. And a merged LoRA model cannot hot-swap tasks in one batch unless you keep the update separate, at some latency cost.

How to answer it in an interview

The spoken structure is: name the method, say what is frozen and trained, give the memory effect, then the inference story and how you would verify quality.

Common mistakes

• "PEFT trains a smaller model" (wrong). It trains a small added piece on top of the full frozen model, which still has to be loaded (right: the frozen weights still occupy memory).
• "All PEFT methods add no inference cost" (wrong). Adapters add sequential layers and prefixes lengthen attention; the LoRA paper shows adapter slow-downs of over 30 percent in short online settings (right: only updates that fold into the weights, such as a merged LoRA update or folded scaling vectors, add nothing).
• "LoRA rank is the number of layers adapted" (wrong). Rank is the width of the narrow dimension of the two matrices (right).
• "PEFT always matches full fine-tuning" (wrong). Results depend on task and model size; prompt tuning only closes the gap at scale (right: say it depends and verify).
• "QLoRA is a different adapter" (wrong). It is LoRA applied on top of a 4-bit quantized frozen model (right).

Follow-up questions

What is the difference between LoRA and full fine-tuning?

Full fine-tuning updates every weight and needs gradients and optimizer states for all of them. LoRA freezes the weights and trains a low-rank update, which in our worked example is 0.06 percent of the parameters and about 13.5 GB against 107.2 GB for weights, gradients and optimizer states, before activations. See the LoRA page for the full treatment.

Does LoRA slow down inference?

Not after merging. You compute $W_0+B·A$ once and run the model unchanged, which the LoRA paper states adds no inference latency by construction. If you keep the update separate to swap tasks, you pay for two small extra matrix products.

What is the difference between prefix tuning and prompt tuning?

Prompt tuning learns vectors only at the input, while prefix tuning learns vectors for the keys and values at every layer. Lester et al. describe prompt tuning as a simplification of prefix tuning, and our worked example shows 81,920 against 5,242,880 parameters for 20 tokens.

How does QLoRA fit a large model on one GPU?

It stores the frozen weights in four bits and trains LoRA matrices in higher precision on top, with paged optimizers for memory spikes. The paper reports finetuning a 65 billion parameter model on a single 48 GB GPU.

What happens if you double the LoRA rank?

The trainable count doubles with it: rank 16 on query and value is 8,388,608 parameters, 0.125 percent of the 6.7 billion¹. The LoRA paper found very small ranks already competitive, so raise the rank only when a held-out set shows the update underfits.

When should you use PEFT instead of full fine-tuning?

When memory, storage or the number of tasks is the constraint, which is the common case for large models. Use full fine-tuning when you have the hardware and a held-out evaluation shows a gap that PEFT does not close, and try prompting first as the fine-tuning versus prompt engineering page suggests.

Sources

1. LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685v2) — arXiv (Hu et al.), 16 Oct 2021
2. Parameter-Efficient Transfer Learning for NLP (arXiv 1902.00751v2) — arXiv (Houlsby et al.), 13 Jun 2019
3. Prefix-Tuning: Optimizing Continuous Prompts for Generation (arXiv 2101.00190) — arXiv (Li and Liang), 1 Jan 2021
4. The Power of Scale for Parameter-Efficient Prompt Tuning (arXiv 2104.08691v2) — arXiv (Lester et al.), 2 Sep 2021
5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv 1910.02054v3) — arXiv (Rajbhandari et al.), 13 May 2020
6. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (arXiv 2205.05638v2) — arXiv (Liu et al.), 26 Aug 2022
7. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314) — arXiv (Dettmers et al.), 23 May 2023
8. LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023
9. PEFT documentation (page shows no date; read 2026-10-24) — Hugging Face, 24 Oct 2026