GPTQ vs AWQ: how do the two four-bit methods differ?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Read this next to the LLM inference optimization guide, which places quantization among the other ways to make serving cheaper.

Why interviewers ask this

The choice of quantizer decides whether a large model fits on the GPUs a team has, so it comes up in serving and cost questions. A 70B-parameter model needs 140 gigabytes for its weights in 16-bit form but about 35 gigabytes at four bits, before overhead (illustrative sizes, computed below).¹

Interviewers use GPTQ vs AWQ to separate candidates who memorize two names from candidates who know what each method changes. A weak answer says "AWQ is newer, so it is better." A strong answer says both are post-training, weight-only methods that need calibration data, names the one idea behind each, and says to benchmark on your own task and hardware. A Staff-level answer adds that speed depends on the kernel and the GPU generation, not only the algorithm, and covers calibration cost and the risk of overfitting the calibration data.

What each one is

GPTQ

GPTQ is a one-shot, post-training weight quantization method that rounds a layer's weights to few bits one column at a time, adjusting the not-yet-rounded weights to absorb each rounding error using approximate second-order information, so the layer's output error stays small.

AWQ

AWQ is a hardware-friendly, low-bit, weight-only quantization method whose full name is Activation-aware Weight Quantization. It finds the small set of weight channels that matter most from the activation distribution and scales them up before rounding, so rounding hurts them less.

Calibration set

A calibration set is a small sample of text that is run through the model during quantization so the method can measure how activations or layer outputs behave. It is data for measuring, not data for training the model again.

GPTQ in plain words

Think of splitting a restaurant bill into items and rounding each price to the nearest ten. If you round the first item down, you can round the next one up to keep the total close. GPTQ does this with weights: after rounding one column of a layer, it nudges the columns not yet rounded so the layer's output stays close to the original. The analogy breaks in one place: GPTQ takes the size of each nudge from the calibration inputs, not from guesswork.

AWQ in plain words

Think of adding a long list of numbers after rounding each to the nearest ten. Each rounding adds a small error, and the errors add up. Now suppose one number in the list is multiplied by a huge factor before the sum. Its rounding error is multiplied too, so that number deserves extra care. Quantization is this problem at scale: the weights are the rounded numbers, and the inputs are the multipliers. AWQ scales the weight with a big multiplier up before rounding, so rounding hurts it relatively less.

The analogy breaks in one place. A model rounds millions of weights in shared groups, not one list. In one precise sentence, GPTQ and AWQ are two ways to choose low-bit weights so that a layer's output barely changes on the calibration data.

Step 1: Both methods calibrate on sample text

GPTQ and AWQ both quantize a trained model after training, keep only the weights in low precision and use a small calibration set to decide how. Weight-only quantization is storing the weights in few bits while the activations stay in 16-bit floating point (fp16). The Hugging Face guide groups both under calibration-based methods and lists good 4-bit accuracy with upfront calibration for them.

Both methods run sample text through the model during calibration. Input: the trained weights and the sample text. Output: low-bit integer weights plus scales, ready to serve. Neither method trains the model again. You now know what the two share: the same inputs and the same kind of output, low-bit integer weights plus per-group scales and zero-points, though in different file formats that a kernel must read specifically. GPTQ and AWQ are alternatives: you apply one or the other, not both. They differ in the decision rule, which the next two steps show.

Step 2: GPTQ rounds each row to minimize error

GPTQ rounds each row of the weight matrix to minimize error, using approximate second-order information about how the layer's output reacts when a weight changes. The GPTQ paper calls it "a new one-shot weight quantization method based on approximate second-order information", and the Transformers documentation says each row of the weight matrix is quantized independently to find the weights that minimize the error.

GPTQ works through a layer one column at a time, in the same order for every row. After it rounds a column, it updates the columns not yet rounded to compensate for that rounding error. The update uses the inverse of the Hessian, H = 2XXᵀ, where X holds the calibration inputs to the layer. The Hessian is the matrix of second derivatives of the layer's squared output error with respect to its weights. This is the Optimal Brain Quantization (OBQ) update made fast with lazy batched updates and a Cholesky factorization (Frantar et al., 2022, §3).

The paper reports quantizing a model with 175 billion parameters in approximately four GPU hours, down to 3 or 4 bits per weight, with negligible accuracy degradation, and still reasonable accuracy at 2 bits or ternary levels (Frantar et al., 2022). Input: a layer's weights and its calibration activations. Output: low-bit integer weights. The paper's full text has a footnote saying the name merges a model-family name with post-training quantization (PTQ).

Step 3: AWQ scales salient channels instead of keeping them in fp16

AWQ protects a small set of important weight channels by scaling them before rounding, not by leaving them in higher precision. The AWQ paper finds that "protecting only 1% salient weights can greatly reduce quantization error", and that to find them "we should refer to the activation distribution, not weights". So the salient weights are not the largest ones: a channel is salient when the inputs it multiplies are large.

AWQ "employs an equivalent transformation to scale the salient weight channels to protect them" (Lin et al., 2023). It multiplies a channel's weights by a factor and divides that channel's inputs by the same factor, so the output is unchanged while the rounding step is relatively smaller for that channel. AWQ scales the salient channels up and quantizes every weight, so all weights end in the same low-bit format. The paper states that AWQ does not rely on any backpropagation or reconstruction. Input: calibration activations. Output: per-channel scales and low-bit weights. In the toy row of the worked example below, scaling one salient channel by 2 cuts the output error from 0.5543 to 0.04.¹

Step 4: A kernel restores fp16 while serving

A quantized checkpoint is smaller in memory, but the arithmetic at serving time still runs in fp16. Four-bit weights are restored to fp16 on the fly during inference: the Transformers documentation says the weights "are quantized to int4, but they're restored to fp16 on the fly". So four-bit weights do not mean four-bit math. The gain is that fewer bytes move from memory, and generating tokens one at a time is usually limited by memory movement; see high bandwidth memory.

The gain is largest when decoding is memory-bound, at small batch sizes; at large batch sizes, or when the whole prompt is processed at once (prefill), the matrix multiplication becomes compute-bound, and the dequantization work can erase the gain or make four-bit weights slower than fp16. The Hugging Face benchmark measured throughput at batch size 1, the favorable case.

A fused kernel does the restoring. The vLLM documentation lists Marlin kernels for both GPTQ and AWQ on Ampere, Ada and Hopper GPUs. The AWQ paper reports that its TinyChat system gives more than 3x speedup over the Hugging Face 16-bit floating point (FP16) implementation on desktop and mobile GPUs (Lin et al., 2023). Speed therefore comes from the kernel and the hardware as much as from the algorithm.

1 · Calibrate2 · GPTQ rounding3 · AWQ scaling4 · ServeReplay

Step-by-step description

1. Sample text runs through the model.
2. Each row is rounded to minimize error.
3. Salient channels are scaled up first.
4. A kernel restores fp16 on the fly.

• Both methods run sample text through the model during calibration.
• GPTQ rounds each row of the weight matrix to minimize error.
• AWQ scales the salient channels up and quantizes every weight.
• GPTQ and AWQ are alternatives: you apply one or the other, not both.
• Four-bit weights are restored to fp16 on the fly during inference.

How GPTQ and AWQ differ side by side

GPTQ and AWQ differ in what they compute during calibration, how long it takes and where they run. GPTQ uses approximate second-order information; AWQ uses activation statistics. On an 8B model one A100 needs about 20 minutes for GPTQ and about 10 for AWQ, according to the Hugging Face selection guide.

The same guide says AWQ often achieves high accuracy at 4-bit and sometimes surpasses GPTQ on specific tasks, and it lists a possible overfit on calibration data as a GPTQ drawback. Both papers report 3-bit results, and the GPTQ paper also reports 2-bit and ternary results (GPTQ paper; AWQ paper). In common tooling, GPTQ offers 2–3 bit widths while AWQ tools target 4-bit (AWQ documentation), and the selection guide notes a noticeable drop below four bits, especially at 2-bit. vLLM supports GPTQ on Volta but not AWQ (as of October 2026); both run on Turing, Ampere, Ada and Hopper (vLLM).

Text description of the diagram

• On an 8B model one A100 needs about 20 minutes for GPTQ and about 10 for AWQ.
• vLLM supports GPTQ on Volta but not AWQ.
• In common tooling, GPTQ offers 2–3 bit widths while AWQ tools target 4-bit.

Worked example

Weights take parameters times bits divided by eight bytes. For a 70B model, 16-bit weights take 140 gigabytes and four-bit weights 35 gigabytes before overhead. With a group size of 128, a 16-bit scale and a four-bit zero-point per group (illustrative storage choices), the overhead is 0.15625 bits per weight, there are 546,875,000 groups, and the total is 36.37 gigabytes, about 3.85 times smaller than 16-bit.¹ That matches the roughly 4x memory saving the Hugging Face guide reports for four-bit methods against bf16, a 16-bit brain floating-point format.

The second calculation shows why scaling a salient channel helps. Take one row of eight weights, 0.90, -0.40, 0.19, 0.27, -0.61, 0.05, 0.33 and -0.22 (illustrative), sharing one scale, with four-bit signed integer levels from -7 to 7. The grid step is 0.9 divided by 7, which is 0.1286. The inputs are 1 everywhere except channel 3, which receives 8. Channel 3 is salient because its input is large, not because its weight is large.¹

The true output is 1.84. Plain rounding turns the 0.19 weight into 0.1286, and the output becomes 1.2857, an error of 0.5543. Now scale channel 3's weight by 2 and divide its input by 2: the weight 0.38 rounds to 0.3857, which is 0.1929 in the original units, and the output is 1.8, an error of 0.04.¹ The largest weight, 0.90, is unchanged, so the grid does not move. Real AWQ derives its scales from activation statistics on the calibration set; this toy row only shows the mechanism. The gain is not this large on every row: it depends on the salient input being large and on where 0.38 lands relative to the grid.

A second toy shows GPTQ's compensation on two weights, 0.34 and 0.64, with a grid step of 0.1 and four calibration inputs (1, 1), (2, 2), (1, 2) and (2, 1) (illustrative). These inputs give H = 2XXᵀ with 20 on the diagonal and 18 off it.¹ Plain rounding gives 0.3 and 0.6, a mean squared output error of 0.0152. GPTQ rounds the first weight to 0.3, leaving an error of 0.04, then moves the second weight by 0.04 × 18 / 20 = 0.036 to 0.676 and rounds it to 0.7. The mean squared output error falls to 0.0022.¹ The compensation works because the two inputs move together, so one weight can make up for its neighbor's error.

Trade-offs and edge cases

When each wins

GPTQ is the usual choice on Volta GPUs or for 3-bit and 2-bit checkpoints. vLLM supports GPTQ on Volta but not AWQ (as of October 2026), and common AWQ tools target 4-bit, so GPTQ-family tooling is the usual route below four bits, where the Hugging Face guide notes a noticeable drop, especially at 2-bit.

On Turing or newer GPUs with a four-bit target, both are supported. Benchmark both on your task when both are supported: AWQ calibrates faster and sometimes surpasses GPTQ on specific tasks, and the same guide says to benchmark accuracy and speed on your specific task and hardware. For a serving design where this choice sits inside a latency budget, see serving a 70B model under a latency target.

Text description of the diagram

• vLLM supports GPTQ on Volta but not AWQ.
• Benchmark both on your task when both are supported.

What breaks or changes

Calibration data matters. The Hugging Face guide lists a possible overfit on calibration data as a GPTQ drawback, while the AWQ paper says AWQ generalizes to different domains and modalities without overfitting the calibration set. Use calibration text that resembles your real traffic.

Tooling moves quickly. As of October 2026 the Transformers documentation says autogptq is no longer supported there and points to the gptqmodel package. Its AWQ page still lists llm-awq, autoawq and optimum-intel, but the autoawq project is no longer maintained and its install note downgrades Transformers to 4.47.1. Prefer the maintained gptqmodel repository, which supports both GPTQ and AWQ workflows, or LLM Compressor, which vLLM recommends for quantizing models.

Weight-only quantization does not by itself shrink the key-value (KV) cache, which grows with context length and batch size. For the wider picture of where memory goes, see GPU fundamentals for machine learning (ML) engineers.

How to answer it in an interview

The spoken structure: say what they share, give the one idea behind each, then name the decision rule (hardware, bit width, benchmark).

Common mistakes

• "AWQ keeps one percent of the weights in fp16." (wrong) AWQ scales the salient channels and quantizes everything. (right) "AWQ protects the salient channels by scaling them, so every weight is stored in the same low-bit format."
• "The salient weights are the largest weights." (wrong) The AWQ paper finds them from the activation distribution. (right) "A channel is salient when its inputs are large."
• "AWQ is always more accurate or faster." (wrong) The Hugging Face guide says AWQ only sometimes surpasses GPTQ, on specific tasks. (right) "I benchmark both on my task, and speed depends on the kernel."
• "Four-bit weights mean four-bit math." (wrong) The weights are restored to fp16 on the fly. (right) "The saving is memory traffic and storage, not cheaper arithmetic."

Follow-up questions

Is AWQ better than GPTQ?

Not universally. The Hugging Face guide says AWQ often achieves high accuracy at 4-bit and sometimes surpasses GPTQ on specific tasks, and it advises benchmarking your own task and hardware.

Is AWQ or GPTQ faster at inference?

It depends on the kernel and the GPU more than on the algorithm. vLLM lists Marlin kernels that serve both formats on Ampere, Ada and Hopper, so on those GPUs the format alone does not decide speed. The gain also shrinks as batch size grows, because large batches and prefill are compute-bound and the dequantization overhead can erase the benefit of four-bit weights.

What do act-order and group size do in GPTQ?

Act-order (desc_act in common toolkits) quantizes columns in order of importance rather than left to right, which usually improves accuracy but can slow some kernels because they need an index to undo the order. A smaller group size follows the weights more closely at the cost of storing more scales, as the memory table above shows.

Can GPTQ or AWQ go below four bits?

Yes in the papers: both report 3-bit results, and GPTQ's authors also report reasonable accuracy at 2 bits or ternary levels (AWQ paper; GPTQ paper). The Hugging Face guide notes a noticeable drop below four bits, especially at 2-bit. In practice the common AWQ toolkits and serving kernels target 4-bit, so GPTQ-family tooling is the usual route for 3-bit or 2-bit checkpoints.

Why does AWQ look at activations instead of weights?

A weight matters more when the inputs it multiplies are large, and only activations show that. The AWQ paper says to refer to the activation distribution, not weights, to identify salient channels.

Which tool should I use to quantize with GPTQ today?

The Transformers documentation says autogptq is no longer supported there and points to gptqmodel; vLLM recommends LLM Compressor for models you will serve with it. Check each page's date, because this tooling changes quickly. The LLM quantization guide covers the wider choices.

Sources

1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv 2210.17323) — arXiv (Frantar, Ashkboos, Hoefler, Alistarh), 31 Oct 2022
2. GPTQ paper, full text (HTML) — arXiv (Frantar et al.), 22 Mar 2023
3. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv 2306.00978) — arXiv (Lin, Tang et al.), 1 Jun 2023
4. Selecting a quantization method — Hugging Face Transformers documentation, 26 Oct 2026
5. GPTQ — Hugging Face Transformers documentation, 26 Oct 2026
6. AWQ — Hugging Face Transformers documentation, 26 Oct 2026
7. Quantization — vLLM documentation (stable), 26 Oct 2026
8. gptqmodel repository — GitHub (ModelCloud), 26 Oct 2026