LLM quantization explained: FP8, INT8, INT4, AWQ, GPTQ

What is LLM quantization?

Quantization maps high precision numbers, usually 16-bit BF16 or FP16 values, to a smaller format such as 8-bit or 4-bit. Each group of values shares a scale that converts the small numbers back to approximately the original range. The model keeps the same architecture and the same number of parameters; each parameter simply takes fewer bytes.

For inference, this matters in two ways. Smaller weights let a model fit on fewer or cheaper GPUs, and they leave more memory for the KV cache, which lets a server run more requests at once. Smaller weights also mean fewer bytes to read per decode step, and decode is often limited by memory bandwidth. The cost is some rounding error, which can reduce answer quality.

Read the notation: W8A8, W4A16, and KV cache precision

Model cards and engine documentation describe a scheme by what it quantizes. W is weights, A is activations, and the number is the bit width. The table shows the common combinations.

Weight-only schemes help the decode phase most, because decode spends its time reading weights. They do little for prefill, which is limited by arithmetic. Schemes that also quantize activations can speed up both, but only on hardware with fast low precision math. LLM inference explained describes the two phases.

Calculate how much memory quantization saves

Weight memory is the parameter count multiplied by the bits per weight, divided by eight. Group-wise formats store one scale per group of weights, which adds a little: a 16-bit scale shared by 128 weights adds 0.125 bits per weight.

GIB = 1024 ** 3


def weight_gib(parameters, bits_per_weight, group_size=None, scale_bits=16):
    """Weight storage only. Group-wise formats add one scale per group."""
    bits = bits_per_weight
    if group_size:
        bits += scale_bits / group_size
    return parameters * bits / 8 / GIB


formats = [
    ("BF16", 16, None),
    ("FP8 or INT8", 8, None),
    ("INT4, group size 128", 4, 128),
]
for parameters, name in ((8e9, "8B"), (70e9, "70B")):
    print(f"{name} parameters")
    for label, bits, group in formats:
        print(f"  {label:<22}{weight_gib(parameters, bits, group):6.1f} GiB")

assert round(weight_gib(8e9, 16), 1) == 14.9
assert round(weight_gib(70e9, 4, 128), 1) == 33.6

8B parameters
  BF16                    14.9 GiB
  FP8 or INT8              7.5 GiB
  INT4, group size 128     3.8 GiB
70B parameters
  BF16                   130.4 GiB
  FP8 or INT8             65.2 GiB
  INT4, group size 128    33.6 GiB

These figures cover weights only. Real checkpoints often keep some layers, such as embeddings or the output head, in 16 bits, and a running server also needs memory for the KV cache, activations, and the engine itself. Use the calculation to decide whether a model can fit, then confirm with the engine’s reported memory usage.

FP8 vs INT8 vs INT4

FP8 is an 8-bit floating point format. The E4M3 variant is usually used for weights and activations in inference. Because it keeps an exponent, it represents values of different sizes better than an 8-bit integer. FP8 Formats for Deep Learning defines the formats. vLLM documents FP8 computation on NVIDIA GPUs with compute capability 8.9 or higher, which covers Ada Lovelace, Hopper, and Blackwell; older GPUs such as Ampere fall back to weight-only FP8. vLLM: FP8 W8A8.

INT8 uses evenly spaced integers with a scale. It is widely supported, including on GPUs without FP8 units, but activations with a few very large values, called outliers, make it hard to quantize activations accurately. LLM.int8() studied these outliers, and SmoothQuant moves part of the difficulty from activations into weights so that W8A8 INT8 keeps its accuracy.

INT4 usually means 4-bit weights with 16-bit activations, W4A16. It gives the largest memory saving of the three and is common for running large models on fewer GPUs, but rounding error is larger, so the method used to choose the 4-bit values matters. Newer GPUs also support 4-bit floating point formats such as NVFP4; check your engine’s hardware table before choosing one.

AWQ vs GPTQ vs SmoothQuant

These names describe how the low precision values are chosen, not a storage format. Each uses a small calibration dataset to measure how the model behaves, then chooses scales or values that keep the output close to the original.

The original papers are GPTQ, AWQ, and SmoothQuant, linked above. In practice, you rarely run these algorithms yourself: many popular models are published in AWQ, GPTQ, or FP8 form, and tools such as LLM Compressor produce new ones. The Hugging Face quantization overview compares the available tools.

Serve a quantized model with vLLM

The simplest path is to serve a checkpoint that was already quantized. Its configuration records the quantization method, and vLLM reads it when loading, so the launch command is the same as for any other model. vLLM can also quantize some formats at load time, such as FP8. The options differ by method and release, so follow the page for your method in vLLM: quantization, which includes a table of supported hardware.

• Prefer a checkpoint published by the model’s authors or a well-known quantizer, and record its exact revision.
• Check that your GPU supports the scheme. A format your hardware cannot compute natively may run slower than BF16.
• Consider quantizing the KV cache separately. It saves memory for long contexts and many requests; vLLM: quantized KV cache explains the options.

For setting up the server itself, follow the vLLM tutorial and swap in the quantized model ID.

Check quality before and after quantizing

A quantized model can look fine in a quick chat and still fail on the tasks you care about. Math, code, long contexts, and non-English text are often the first to degrade. Compare against the 16-bit baseline on the same prompts with the same sampling settings.

1. Collect a few hundred prompts from your real traffic, with expected answers or grading rules.
2. Run the 16-bit model and the quantized model with greedy decoding and identical prompts, then score both.
3. Add a standard benchmark for your task type, such as a math or coding suite, to catch broad regressions.
4. Measure speed and memory at your real concurrency. LLM inference metrics explains which numbers to record.
5. Accept the quantized model only if both the quality gap and the speed gain meet targets you set in advance.

LLM quantization FAQ

Does quantization make a model faster? Usually, when the format matches the hardware. Weight-only formats speed up memory-bound decode. W8A8 formats can also speed up prefill and large batches. An unsupported format can be slower than BF16.

Is 4-bit good enough? For many chat and retrieval tasks, a well-made 4-bit AWQ or GPTQ model is close to the original. For precise reasoning or code, test carefully, because the gap is often larger.

Should I choose FP8 or INT8? On GPUs with FP8 support, FP8 is usually the easier choice because it handles outliers better. On older GPUs, INT8 or a weight-only format is the practical option.

Is quantizing the KV cache the same as quantizing weights? No. It is a separate setting that shrinks the cache for long contexts and many requests. It can be combined with weight quantization.