What is LoRA fine-tuning, and how is QLoRA different?

In this article

1. Before you start
2. Why interviewers ask this
3. LoRA from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the fine-tuning and large language model (LLM) training learning path.

Why interviewers ask this

LoRA is a standard way to adapt a large model without owning a cluster, so it comes up when the conversation turns to fine-tuning in practice. The question separates candidates who can recite "it trains fewer parameters" from candidates who can say what is trained, what is frozen, what the memory bill really is, and what happens at inference.

A weak answer stops at "low-rank matrices". A strong answer writes the update as a product of two thin matrices, counts the parameters, says why the weights still have to fit in memory, explains that the update can be merged away for serving, and then explains what QLoRA changes. The worked example below does the counting on a real model configuration.

LoRA from first principles

QLoRA

QLoRA is a fine-tuning method that stores the frozen base model in 4-bit precision and trains LoRA adapters on top of it in higher precision, so a very large model fits on a single graphics processing unit (GPU).

NF4

NF4, short for NormalFloat 4, is a 4-bit data type that stores weights using levels placed to suit normally distributed values, which lets QLoRA keep a frozen base model in very little memory with small quantization error.

In plain words

Think of a multiplication table. It has hundreds of cells, yet one column and one row describe it all, because each cell is a column entry times a row entry. A grid that can be described that way has rank 1. A grid built by adding r column-and-row pairs has rank at most r. LoRA bets that the change a task needs is like this. A big grid of changes can be described by a few pairs. The analogy breaks in one place: LoRA does not fix the pairs in advance, because training finds them.

In one precise sentence: LoRA keeps the original weight matrix W frozen and learns an update B times A, where B is tall and thin, A is short and wide, and the shared inner size is the rank r.

Step 1: Full fine-tuning changes every weight

In full fine-tuning the training loop updates every weight of the model. Each weight needs a stored gradient and two running statistics for the Adam optimizer, on top of the weight itself. Adam is the optimizer that turns gradients into updates using those two statistics. Mixed precision is training with 16-bit weights and gradients plus 32-bit copies kept for accuracy.

The ZeRO paper counts the cost for mixed-precision training with Adam: 2 bytes for the 16-bit weight, 2 bytes for its gradient and 12 bytes of optimizer state (a 32-bit copy of the weight plus the two Adam statistics, 4 bytes each), 16 bytes per parameter in total (Rajbhandari et al., 2019). For a model of 6.7 billion parameters that is about 108 gigabytes (GB) before any activations (the intermediate results kept for the backward pass) are stored.¹ The input is a model and a dataset; the output is a full second copy of every weight. This is the bill LoRA is designed to avoid.

Step 2: LoRA freezes W and learns a low-rank update

LoRA leaves the original matrix W untouched and adds a trainable path beside it. The layer's output becomes W times x plus a scaled B times A times x. Matrix A has r rows and as many columns as W has, matrix B has as many rows as W and r columns, so their product has the shape of W but rank at most r.

The LoRA paper starts A from a random Gaussian and B from zero, so the update B A is zero at the start and the model begins exactly as the pre-trained one (Hu et al., 2021, section 4.1). The update is scaled by alpha divided by r, where alpha is a constant; the parameter-efficient fine-tuning (PEFT) documentation gives this as the default scaling (as of October 2026).

Step 3: Count the trainable parameters

A full matrix with d_in inputs and d_out outputs has d_in times d_out entries. The LoRA pair has r times (d_in plus d_out). In the Llama 2 7B model the attention matrices are 4,096 by 4,096, so one matrix has 16,777,216 entries while a rank-8 pair has 8 times 8,192, which is 65,536: 256 times fewer for that matrix.¹ Applying rank 8 to the query and value matrices of all 32 layers gives 32 layers times 2 matrices times 65,536, which is 4,194,304 trainable parameters out of 6,738,415,616, or 0.0622%.¹ The count grows in direct proportion to r: rank 4 trains 2,097,152 parameters, rank 16 trains 8,388,608 and rank 64 trains 33,554,432.¹

Text description of the diagram

• Rank 4 trains 2,097,152 parameters, rank 16 trains 8,388,608 and rank 64 trains 33,554,432.
• Rank 8 on the query and value matrices of all 32 layers is 4,194,304 trainable parameters.

Step 4: Memory falls by less than the parameter count suggests

The frozen weights still have to sit in graphics processing unit (GPU) memory. With 16-bit weights the 6.7-billion-parameter model needs 13.5 GB just to be loaded, and that does not change when only 0.06% of the parameters train.¹ What LoRA removes is the gradient and optimizer state of the frozen weights: the trainable part needs 16 bytes per parameter, which for the adapter is about 67 megabytes.

The total, weights plus adapter state, is about 13.5 GB against 108 GB for full fine-tuning, an eightfold saving in model-state memory, not a 1,600-fold one.¹ The LoRA paper itself reports a 3-times reduction in GPU memory and a 10,000-times reduction in trainable parameters for the 175-billion-parameter generative pre-trained transformer (GPT) 3 with Adam, which shows the same gap between the two numbers (Hu et al., 2021, abstract).

Step 5: Merge the update for inference

After training, the pair can be folded into the original weight: the new matrix is W plus alpha over r times B A. The LoRA paper notes that this means no additional inference latency, and that W can be recovered by subtraction to switch tasks (Hu et al., 2021, section 4.1). The PEFT documentation exposes this as a merge call that returns a standalone model, and it also lets you keep several small adapters on top of one base model (as of October 2026). A check on a tiny random matrix confirmed the merged weight gives the same output as the two-path form to within 1e-12.¹

Step 6: QLoRA stores the frozen base in 4 bits

QLoRA keeps the idea and shrinks the frozen part. Dettmers et al. (2023) backpropagate through a frozen quantized model, one whose weights are stored with fewer bits per number, 4 instead of 16, into LoRA adapters, using a 4-bit data type called NormalFloat (NF4) designed for normally distributed weights, double quantization (quantizing the quantization constants themselves) and paged optimizers that absorb memory spikes. Backpropagation is the step that sends the error backwards through the layers to compute each gradient.

They report fine-tuning a 65-billion-parameter model on one 48 GB GPU with quality comparable to full 16-bit fine-tuning, and a model family that reaches 99.3% of ChatGPT's score on the Vicuna benchmark (a set of chat prompts whose answers a language model judges) after 24 hours on one GPU. At 4 bits, half a byte per weight, the 6.7-billion-parameter model's weights take about 3.4 GB, and 65 billion parameters take about 32.5 GB, consistent with that claim.¹

1 · Full tuning2 · Two paths3 · Count4 · Memory5 · Merge6 · QLoRAReplay

Step-by-step description

1. Full fine-tuning: x goes through W, and every weight in W is trained, at 16 bytes each.
2. LoRA adds a second path beside W: x goes through A then B, is scaled by alpha over r, and is added to W's output. W itself is not trained.
3. Rank 8: A and B hold 32,768 + 32,768 = 65,536 numbers, the only trained weights.
4. W is still loaded; only its gradients and optimizer state are gone.
5. Folding in the update gives W' = W plus alpha over r times BA, which serves with no extra latency.
6. The frozen W is stored in 4 bits; A and B stay 16-bit.

• Full fine-tuning needs 16 bytes per parameter for model states.
• A rank-8 pair for a 4,096 by 4,096 matrix has 65,536 parameters.
• The frozen weights still have to sit in GPU memory.
• The merged weight W plus alpha over r times BA adds no inference latency.
• QLoRA stores the frozen W in 4 bits and keeps A and B in 16 bits.

Worked example

Everything below uses the Llama 2 7B configuration: hidden size 4,096, feed-forward size 11,008, 32 layers and a vocabulary of 32,000 (config.json, as of October 2026). Counting the embeddings, the four attention matrices and three feed-forward matrices per layer, the normalisation weights and the output head gives 6,738,415,616 parameters.¹

The memory rows count model states only. Activations, which grow with batch size and sequence length, come on top, and the 4-bit figure ignores the small overhead of the quantization constants, so real numbers are higher.¹

Text description of the diagram

• Full fine-tuning model states take 107.8 GB.
• LoRA needs 13.5 GB for the weights plus adapter state.
• QLoRA needs 3.4 GB with 4-bit weights, idealised.

Trade-offs and edge cases

LoRA trades some flexibility for a very small training footprint, and four choices decide how well it works.

The rank. A larger rank gives the update more capacity and costs proportionally more parameters. The LoRA paper reports that a rank of 1 suffices for many tasks on GPT-3 175B, while on GPT-2 Medium the end-to-end generation results peak at rank 16 for validation loss and rank 4 for the bilingual evaluation understudy (BLEU) score (Hu et al., 2021, Table 6 and Table 18). The QLoRA paper, which tuned all linear layers, found that the rank r barely mattered once every layer was adapted (Dettmers et al., 2023, appendix A). So rank is a setting to validate, not a lever that reliably buys quality.

Which matrices. The LoRA paper adapted only the query and value matrices in most experiments (Hu et al., 2021, section 5.1) and found that pair worked best at a fixed parameter budget (section 7.1), and the PEFT guide says LoRA can be applied to any subset of weight matrices but is typically applied to the attention blocks (as of October 2026). The QLoRA paper reports that adapting only query and value does not match full fine-tuning on large models, and that adapting all linear layers is what recovers it (Dettmers et al., 2023, section 4). All seven linear matrices per layer at rank 16 train 39,976,960 parameters, 0.5933% of the model, against 0.0622% for rank 8 on two matrices.¹

The scaling. Alpha over r sets how strongly the update counts. The PEFT documentation also describes a rank-stabilised variant that divides alpha by the square root of r instead (as of October 2026).

When LoRA is the wrong choice. Reported practice differs from the headline claims. Biderman et al. (2024) compared LoRA with full fine-tuning on code and mathematics and found that LoRA substantially underperforms there, while it forgets less of what the model knew outside the target domain; they also found that full fine-tuning learns weight updates with a rank 10 to 100 times higher than typical LoRA settings (abstract, read 2026-10-24).

If the task needs a large shift in knowledge or skill, such as new code or maths ability, and you can afford the memory, full fine-tuning is the safer baseline; if the task is a style, format or light-domain adjustment, or memory is the constraint, LoRA is the usual first choice.

What QLoRA risks. Storing the base model in 4 bits adds quantization error, and the NF4 weights are dequantized to 16-bit brain floating point (BF16) for every forward and backward pass (Dettmers et al., 2023, section 3), which adds compute compared with 16-bit LoRA. The authors report no loss of runtime or predictive performance against a 16-bit full fine-tune, but measure throughput and quality on your own task before trusting the paper's headline comparison. For number formats, see 16-bit number formats, and for the memory arithmetic of serving, the GPU fundamentals page. The attention matrices LoRA usually targets are explained in the transformer overview, and a page on multi-head attention covers how they split into heads: multi-head attention.

How to answer it in an interview

The spoken structure: say what is frozen and what is trained, give one parameter count, say what memory still costs, then say what QLoRA changes.

Common mistakes

• "LoRA makes the model 10,000 times smaller in memory." (wrong) It cuts trainable parameters, not the frozen weights that must be loaded. (right) "It cuts optimizer and gradient memory; the weights still need to fit."
• "LoRA slows down inference." (wrong) The update can be merged into the weight, so the merged model runs like the original. (right) "Unmerged adapters add a small cost; merged ones add none."
• "A higher rank is always better." (wrong) Parameters grow linearly with rank, and the paper shows small ranks can suffice. (right) "I choose the rank by validating a few values."
• "QLoRA trains the model in 4 bits." (wrong) Only the frozen base is stored in 4 bits; the adapters are trained in higher precision. (right) "The gradients flow through a 4-bit base into 16-bit adapters."

Follow-up questions

What is LoRA and QLoRA?

LoRA freezes a model and trains a small low-rank update beside chosen weight matrices; QLoRA does the same on a frozen base stored in 4 bits. In short, LoRA saves optimizer memory and QLoRA also saves weight memory.

What is QLoRA?

QLoRA is LoRA on top of a frozen base model stored in 4-bit precision, using the NormalFloat 4-bit data type, double quantization and paged optimizers. Its authors report fine-tuning a 65-billion-parameter model on a single 48 GB GPU.

How do you use QLoRA?

Load the base model in 4-bit form, attach LoRA adapters with a chosen rank, scaling and target matrices, train only the adapters, and afterwards either keep the adapter file separate or merge it into a higher-precision copy of the model. Libraries such as the Hugging Face PEFT package expose rank, alpha and target modules as settings. For the production side, see the machine learning operations hub and the deep learning hub.

What rank should you choose for LoRA?

Start small, for example 8, and validate a few values on your own task. The LoRA paper reports that rank 1 sufficed for many GPT-3 tasks while other models peaked at rank 4 or 16, and the QLoRA paper found that rank barely mattered once all linear layers were adapted (Dettmers et al., 2023, appendix A), so there is no universal value and the choice of layers matters more.

Is LoRA the same as fine-tuning?

LoRA is a way of fine-tuning, not a different thing: it continues training a pre-trained model on your data but updates only a small low-rank add-on while the original weights stay frozen. Unlike full fine-tuning, which changes every weight, the result is a small adapter file, and once merged the model serves with no extra latency.

Which weight matrices should LoRA adapt?

The LoRA paper adapted only the query and value matrices in most experiments (Hu et al., 2021, section 5.1), and found that pair worked best at a fixed parameter budget (section 7.1), and the PEFT guide says attention blocks are the typical choice. The QLoRA paper found that this does not match full fine-tuning on large models and that all linear layers are needed (Dettmers et al., 2023, section 4). The cost is modest: 0.0622% of the 7B model for two attention matrices at rank 8 versus 0.5933% for all seven linear matrices at rank 16.¹

Sources

1. LoRA: Low-Rank Adaptation of Large Language Models — arXiv (Hu et al.), 17 Jun 2021
2. QLoRA: Efficient Finetuning of Quantized LLMs — arXiv (Dettmers et al.), 23 May 2023
3. LoRA (PEFT conceptual guide) — Hugging Face (date = read date), 24 Oct 2026
4. LoRA Learns Less and Forgets Less — arXiv (Biderman et al.), 15 May 2024
5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — arXiv (Rajbhandari et al.), 4 Oct 2019
6. Llama-2-7b-hf config.json (mirror of the model configuration) — Hugging Face (date = read date), 24 Oct 2026