What is LLM quantization, and how much memory does it save?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM quantization from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization learning path.

Why interviewers ask this

Quantization is the most common first answer to "this model does not fit on our GPU", so the question tests whether you can turn a memory problem into arithmetic and then say what it costs in accuracy. Interviewers use it to separate candidates who say "make the numbers smaller" from candidates who can explain scales, outliers and how to check the damage.

A weak answer lists names such as 8-bit integer (INT8) and 4-bit integer (INT4). A strong answer gives the memory saved, says why a few large values cause most of the error, names one method that handles them, and says how to validate on the real task.

LLM quantization from first principles

Quantization

Quantization is a compression method that stores each number of a model as a low-bit integer (or a low-bit floating-point code) plus a shared scale instead of a 16-bit float, trading a small rounding error for much less memory.

Scale

A scale is a stored positive number that converts a low-bit integer back to the size of the original value; the quantized weight times the scale approximates the original weight.

Post-training quantization

Post-training quantization (PTQ) is a form of quantization that is applied to an already trained model using at most a small calibration set and no full retraining.

In plain words

Think of storing every length as two digits times a step. For pieces under 99 cm the step is 1 cm, so each length is off by half a centimetre at most. Add one 99 m rope and the step must grow to 1 m, so a 12 cm piece rounds to 0. One big value spoils the precision of every value that shares its step. The analogy breaks in one place. A model does not round once for a human to read; the rounding error flows through billions of multiplications.

In one precise sentence: quantization keeps a small integer per weight plus a scale, and rebuilds approximate weights when the model runs. Jacob et al. (2017) describe integer-only inference and pair it with a training procedure that preserves accuracy after quantization, now called quantization-aware training (QAT), which trains the model while simulating the rounding.

Step 1: Weights are numbers with a fixed bit budget

Every weight is stored in a fixed number of bits, so a model's size is its parameter count times bits per weight. A 70-billion-parameter model in 16-bit needs 70 billion times 2 bytes, which is 140 GB for the weights alone.¹

Cutting to 8 bits halves that and cutting to 4 bits quarters it, and the saving is in memory traffic as well as storage, because decoding reads the weights from memory at every step. The input to this step is a trained 16-bit model; the output is a memory target. Real formats add a small overhead for scales: with one 16-bit scale per 128 weights, 8-bit costs 8.125 bits per weight and 4-bit costs 4.125, giving 71.1 GB and 36.1 GB.¹ You now know the prize. Next comes how the rounding works.

Step 2: A scale turns floats into integers

Absmax quantization finds the largest absolute value in a block of weights, sets the scale to that value divided by the largest integer, divides every weight by the scale and rounds. For 8-bit signed integers the largest integer is 127.¹

Take eight weights: 0.12, -0.45, 0.30, 0.02, -0.08, 0.51, -0.33 and 0.19. The largest absolute value is 0.51, so the scale is 0.51 divided by 127, about 0.004016. The integers are 30, -112, 75, 5, -20, 127, -82 and 47. Multiplying back gives 0.1205, -0.4498 and so on, and the largest error is 0.0013.¹ The input is a block of floats; the output is integers plus one scale. Each weight moves by at most half a step of the scale. This is symmetric quantization; asymmetric schemes add a stored offset called a zero point, and a per-channel scale means one scale for each row of a weight matrix.

Step 3: Outliers and group scales decide the error

A single large value sets the shared scale, so it coarsens all the others. In an illustrative test of 16,384 random weights with a standard deviation of 1 (a measure of their typical size), 8-bit absmax with one scale for the whole tensor has a root-mean-square (RMS) error of 0.0091, and with one scale per group of 128 weights 0.0064.¹ Replace one weight with the value 40, which is 40 standard deviations (the next largest is 4.02), and the 8-bit per-tensor error rises to 0.0905, ten times worse. The group-of-128 error is 0.0100, because the outlier spoils only its own group.¹

At 4 bits with group scales the error is already 0.118 without an outlier, eighteen times the 8-bit 0.0064; the outlier moves it only to 0.147, while per tensor it goes 0.166 to 0.989.¹ At 8 bits the outlier dominates.

Real models have outliers mostly in activations; Dettmers et al. (2022) keep such feature dimensions in 16-bit while computing more than 99.9% of values in 8-bit, and SmoothQuant moves that difficulty into the weights.

Step 4: Smarter methods choose better scales

Methods differ in how they treat values that quantize badly. GPTQ is named for the generative pre-trained transformer (GPT) models it quantizes. GPTQ is a one-shot weight-only method that uses approximate second-order information to choose roundings, quantizing a 175-billion-parameter model in about four GPU hours to 3 or 4 bits with negligible accuracy degradation, as its authors report.

Activation-aware weight quantization (AWQ) observes that not all weights matter equally: protecting about 1% of salient weights, whose channels are found from activation statistics, substantially reduces error, and it scales those channels up instead of keeping them in a different format. SmoothQuant quantizes both weights and activations to 8 bits by moving difficulty from activations to weights, and reports up to 1.56 times speedup and 2 times less memory. QLoRA (quantized low-rank adaptation) trains small add-on matrices on top of a frozen 4-bit model so that a 65-billion-parameter model fits on one 48 GB GPU; it stores the weights in 4-bit NormalFloat (NF4), a code rather than a plain integer.

1 · 16-bit weights2 · Scale and round3 · Group scales4 · Smarter methodsReplay

Step-by-step description

1. A 70B model needs 140 GB in 16-bit.
2. Divide by a scale and round to integers.
3. One scale per 128 weights confines outliers.
4. GPTQ and AWQ choose scales and rounding.

• A 70-billion-parameter model needs 140 GB for its weights in 16-bit.
• Absmax quantization divides each weight by a scale and rounds it to an integer.
• One scale per group of 128 weights confines the damage of an outlier.
• GPTQ and AWQ choose the rounding and the scales more carefully.

Worked example

The numbers below come from one script. A 70-billion-parameter model has 140 GB of weights in 16-bit, 71.1 GB in 8-bit and 36.1 GB in 4-bit when one 16-bit scale per 128 weights is counted, so the 4-bit model fits in the memory of one 80 GB GPU with room for a cache, while the 16-bit model needs two.¹ The error figures use 16,384 random weights (the outlier is the value 40 in place of one weight) and symmetric absmax rounding (4-bit uses the integers -7 to 7); they are illustrative, not measurements of a real model.¹

Text description of the diagram

• Weights of a 70-billion-parameter model take 140 GB in 16-bit.
• They take 71.1 GB in 8-bit and 36.1 GB in 4-bit with one 16-bit scale per 128 weights.

Text description of the diagram

• With no outlier, 8-bit per-tensor error is 0.0091 and 8-bit group-of-128 error is 0.0064.
• With one weight set to 40 (40 standard deviations), 8-bit per-tensor error rises to 0.0905 while group-of-128 error is 0.0100.

Trade-offs and edge cases

Quantization trades accuracy and engineering effort for memory and speed, and the trade is not the same for every part of the model. Five cases matter.

First, weight-only versus weights and activations. Weight-only methods such as GPTQ and AWQ shrink memory and the weight reads; quantizing activations too, as SmoothQuant and LLM.int8() do, lets the matrix multiplication itself run in integers. Second, the cache: the KV cache is a separate memory bill, described in the KV cache page, and a smaller model leaves room for more concurrent requests in continuous batching.

Third, hardware: the speed you gain depends on whether the GPU has fast integer or low-bit paths, so measure on your hardware. Fourth, the number format: 8-bit floating point (FP8), proposed by Micikevicius et al. (2022), is a format with two encodings, one with 4 exponent bits and 3 mantissa bits (E4M3) and one with 5 exponent bits and 2 mantissa bits (E5M2), and its authors also examine FP8 post-training quantization; the approach called microscaling pairs a per-block scaling factor with narrow floating-point or integer elements.

As of October 2026, NVIDIA's technical blog (June 2025) states that FP8 is supported by Hopper and Blackwell GPUs, and that Blackwell Tensor Cores support three 4-bit floating-point (FP4) formats: plain FP4, MXFP4 (a microscaling format with one shared scale per 32 values) and NVIDIA's own 4-bit format (NVFP4). NVFP4 stores one FP8 scale for every 16 values, which is the group-scale idea of Step 3 applied to a 4-bit float, at 4.5 bits per value according to that post. Check which formats your GPU and serving software support before choosing. Fifth, the method claims in the table below come from each paper's own tests on its own models, so always check your task.

How to answer it in an interview

The spoken structure: say what is stored, give the memory arithmetic, name the outlier problem, then say how you would validate.

Common mistakes

• "Quantization just rounds everything the same way." (wrong) One outlier can stretch a shared scale and ruin precision for the rest. (right) "Per-group scales confine outliers."
• "4-bit is always four times faster." (wrong) Memory falls fourfold but speed depends on hardware paths and on activations. (right) "Memory falls, and speed must be measured."
• "Quantizing the weights solves the memory problem." (wrong) The KV cache is a separate memory cost. (right) "I size the weights and the cache separately."
• "Perplexity is enough to validate." (wrong) A model can keep perplexity and still fail on your task. (right) "I test the real task and its failure cases."

Follow-up questions

What is quantization used for?

Quantization is used to make a trained model cheaper to run: it fits larger models on fewer or smaller GPUs, lowers memory traffic during decoding, and raises how many requests one machine serves. GPTQ, for example, lets a 175-billion-parameter model run generative inference on a single GPU, according to its authors.

Which LLM quantization is best?

No method is best everywhere. LLM.int8() is a safe 8-bit choice because it keeps outliers in 16-bit (Dettmers et al., 2022); GPTQ and AWQ target 3 to 4 bits for weights only; SmoothQuant targets 8-bit weights and activations. Choose by target hardware and accuracy budget, then test on your task.

LLM size vs quantization?

A bigger model quantized to fewer bits and a smaller model in 16-bit can need the same memory. Four-bit weights for 70 billion parameters take 36.1 GB, about what 18 billion parameters take in 16-bit (36.1 GB divided by 2 bytes is 18.05 billion).¹ Which is better is empirical, so compare both on your task at your memory budget.

What is the difference between post-training quantization and quantization-aware training?

Post-training quantization changes a finished model using a small calibration set. Quantization-aware training simulates the rounding while training, as described by Jacob et al. (2017), which usually keeps accuracy better at higher cost.

You quantized your model and accuracy dropped: what do you do?

Find where the loss comes from: check the outlier layers, move to per-group scales, keep sensitive layers in 16-bit, or switch to a method such as AWQ or GPTQ. Recompute your task metric after each change. The machine learning operations hub and the deep learning hub cover the neighbouring topics, and the transformer overview explains the layers involved.

Sources

1. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — arXiv (Dettmers et al.), 15 Aug 2022
2. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv (Frantar et al.), 31 Oct 2022
3. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — arXiv (Lin et al.), 1 Jun 2023
4. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — arXiv (Xiao et al.), 18 Nov 2022
5. QLoRA: Efficient Finetuning of Quantized LLMs — arXiv (Dettmers et al.), 23 May 2023
6. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — arXiv (Jacob et al.), 15 Dec 2017
7. FP8 Formats for Deep Learning — arXiv (Micikevicius et al.), 12 Sep 2022
8. Microscaling Data Formats for Deep Learning — arXiv (Rouhani et al.), 16 Oct 2023
9. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference — NVIDIA Technical Blog, 24 Jun 2025