---
title: "What does quantization trade away?"
description: "Quantization rounds an AI model's numbers to fewer bits. See how much memory and speed that buys, and what it costs in accuracy."
topic: "AI systems"
chapter: "inside-the-chip"
order: 20
minutes: { plain: 6, deep: 22 }
prerequisites: ["How much memory an AI model needs (lesson 18)", "Why a chip waits on memory (lesson 19)", "Reading short Python functions"]
seo_title: "What does LLM quantization cost in accuracy?"
date: 2026-10-19
---
# What does quantization trade away?
Llama 3.1 8B, an open large language model (LLM), stores 8,030,261,248 numbers called weights. At 16 bits each (a bit is one 0 or 1) the file is 14.96 GiB (a GiB is a little over a billion bytes, each 8 bits). Rounding most of the weights to 4 bits cuts it to 5.31 GiB and lifts the speed limit on one NVIDIA H100 graphics card from 223 to 721 tokens a second (a token is a chunk of text, often a short word). This lesson shows what the rounding costs and why one huge weight can ruin it.
## In one minute
LLM quantization costs some accuracy in return for a smaller model that runs faster. It rounds the model's weights so each takes fewer bits. Think of a menu priced to the cent. Round every price to the nearest dollar and a ten-dish bill barely moves, because some prices round up and others down. Round to the nearest ten dollars and the bill drifts. A model behaves the same way: 8 or 4 bits change its answers a little, and 3 bits change them more. Fewer bits also mean fewer bytes to store and to read for every token.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/quantization/widget.html). It rounds 16 sample weights to 16, 8, 4 or 3 bits and shows the size of Llama 3.1 8B, its speed limit and the error: how far rounded weights sit from the originals. It opens at 4 bits with one scale per group of four weights. Tick "Add one big weight" and the error jumps from 4.7% to 19.1%; choose "One for all" and it reaches 34.7%.
## How it works
### Rounding to a grid
Each weight is rounded onto an evenly spaced grid. A quantization scale, one stored number for a run of neighbouring weights, sets the step between grid points. At 4 bits a weight is kept as a whole number from −7 to 7, and the model multiplies it by the scale when it uses the weight. The scale is chosen so the largest weight in the run lands on the last step. Taking the closest step is called round-to-nearest.
Figure 1 rounds 0.32 in a run whose largest weight is 0.70. At 4 bits the steps are 0.1 apart and 0.32 becomes 0.30. At 3 bits they are 0.23 apart and 0.32 becomes 0.23. Fewer bits mean wider steps and bigger errors.
*Figure 1. One weight rounded on a 4-bit and a 3-bit grid. Both reach the run's largest weight, 0.70.*
### Why scales come in groups
A quantization group is a run of weights that share one scale. The scale follows the group's largest weight, so if one weight is far larger than the rest, the steps stretch to reach it and ordinary weights round to zero.
Figure 2 uses a made-up matrix (a grid of numbers) of 262,144 weights at 4 bits. Error is the typical gap between a weight and its rounded value, as a share of a typical weight. With one scale for the whole matrix, setting one weight to 1.0, fifty times a typical weight, lifts the error from 19.5% to 99.5%. With a scale for each group of 128 weights, the damage stays in one group: 11.8% to 11.9%.
Each 16-bit scale is shared by 128 weights, which adds 0.125 bits per weight: 4 bits become 4.125.
*Figure 2. Error at 4 bits on a made-up matrix, without and with one weight set to 1.0.*
### Smaller and faster
Figure 3 counts Llama 3.1 8B at four bit widths, scales included. The token table (which turns tokens into numbers) and the output matrix (which scores the next token) stay at 16 bits, as in published files. Bits alone would give 3.74 GiB at 4 bits; the scales and the 16-bit parts add the rest.
Speed follows from [Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/). To write a token the card reads the weights from its memory, so memory speed (bandwidth) divided by the bytes read per token caps the tokens a second for one user. An H100 reads 3.35 terabytes (trillions of bytes) a second, so the cap is 223 tokens a second at 16 bits and 721 at 4 bits, 3.2 times more. It is a ceiling, not a forecast.
*Figure 3. Llama 3.1 8B with a 16-bit scale per 128 weights, and the one-user speed limit on a socketed (SXM) H100 card (3.35 TB/s). The limit comes from bandwidth alone.*
### What it costs in quality
The widget's error is a proxy: it measures how far rounded numbers sit from the originals, not whether answers change. The real test runs the finished model. Perplexity is one such test: a score of how surprised a model is by real text, where lower is better.
One study rounded three models in groups of 128 with plain rounding. Perplexity rose 2.0% to 4.8% at 4 bits and 13.1% to 21.8% at 3 bits (Figure 4). Methods that round with care do better: at 3 bits one reached 6.24 on Llama-2 7B where plain rounding gave 6.66 (16 bits: 5.47). Two published copies of Llama 3.1 8B Instruct (its chat version), both made with one such method, GPTQ, each tested against its own 16-bit run, scored an average of benchmarks (fixed test sets) 74.3 to 73.5 at 4 bits and 74.05 to 74.12 at 8.
*Figure 4. Rise in perplexity on WikiText-2 (a standard Wikipedia text sample) over 16 bits, round-to-nearest in groups of 128. Source: Lin et al., AWQ (2023), Table 4.*
## What this means when you build
- **Try 8 bits first, then 4.** The 8-bit GPTQ copy lost nothing visible on the average score and the 4-bit GPTQ copy lost 0.8 points. Test any copy on your own task.
- **Read a score by its parts.** On that 4-bit copy a maths test (GSM-8K) held at 82.9 against 82.8, while a harder test (MMLU-Pro) fell from 30.8 to 28.8. An average hides which skill dropped.
- **Budget the file with scales and kept weights.** At 4 bits the 8B model is 5.31 GiB, not the 3.74 GiB that half a byte per weight suggests. [How much memory does an AI model need?](https://spicyshark.co/learn/model-memory/) adds the attention cache (a chat's notes) and the rest of the bill.
- **Expect a ceiling, not a speed.** Going from 16 bits to 4 raises the one-user limit by at most 3.2 times on the same card. A real server also reads the attention cache and serves many users.
- **Go below 4 bits with care.** At 3 bits, plain rounding raised perplexity by 13% to 22%. Ask which method made a 3-bit file.
## Go deeper
*Optional. For readers who write Python. The code is in the `code/` folder; every number below was printed by `python3 show_numbers.py`, worked out by hand, or comes from a cited source read on 2026-10-19.*
### The naive approach: one scale for the whole matrix
The simplest quantizer finds the largest weight in the matrix, divides it by the largest whole number that fits in the bits, and rounds everything on that grid:
```python
def naive(w, bits):
scale = np.abs(w).max() / (2 ** (bits - 1) - 1)
return np.rint(w / scale) * scale
```
The tables below come from the full version in `quantize.py`, which also stores the scale as a 16-bit float. It ran on a seeded matrix of 256 x 1,024 bell-curve weights with standard deviation 0.02 (the `initializer_range` of the Llama config; a stand-in, not a trained matrix). The largest weight is 0.0946, 4.7 standard deviations. Error is the root-mean-square gap between the weights and their rounded versions, divided by the root-mean-square size of the weights:
| bits | one scale | one per 128 |
|---|---|---|
| 8 | 1.1% | 0.6% |
| 4 | 19.5% | 11.8% |
| 3 | 45.5% | 27.5% |
Groups already help on ordinary weights, because each group's largest weight is smaller than the matrix's. The gap opens when one weight is large. Here is the same matrix at 4 bits with its top-left weight set to the value in the first column. The standard deviations are the weight divided by 0.02:
| that weight | one scale | one per 128 |
|---|---|---|
| none | 19.5% | 11.8% |
| 0.1 (5 std) | 20.6% | 11.8% |
| 0.2 (10 std) | 41.2% | 11.8% |
| 0.5 (25 std) | 88.6% | 11.9% |
| 1.0 (50 std) | 99.5% | 11.9% |
With 1.0 the one scale is 1.0 / 7 = 0.143, seven times a typical weight, so only 98 of the 262,144 weights keep a nonzero code. The 50 standard deviations are an illustration, not a measurement. At 8 bits the same weight gives 11.3% with one scale and 0.7% per group. Without any large weight, 4 bits send 26.3% of the weights to zero with one scale and 16.0% with groups; 3 bits send 56.9% and 36.2%.
### The method, in formulas
```text
qmax = 2^(bits - 1) - 1 (127 at 8 bits, 7 at 4, 3 at 3)
scale = largest |w| in the group / qmax (stored as a 16-bit float)
code = round(w / scale), ties to even, kept inside -qmax..qmax
w_hat = code x scale
size = embeddings x 2 bytes + others x bits / 8 + others / 128 x 2 bytes
tokens a second <= bandwidth / bytes read for one token
```
This is round-to-nearest with a symmetric (absmax) scale. The most negative code, −2^(bits−1), goes unused so that the grid is symmetric around zero. Ties round to even, as `numpy.rint` does, because JavaScript's `Math.round` rounds them up and the widget must give the same numbers as the Python. The code is also clipped to the range as a safeguard, because a 16-bit scale can round down slightly. A group of zeros gets a scale of zero and codes of zero.
### A hand-worked example
One group of four weights, `[0.32, -0.10, 0.04, -0.70]`.
1. The largest size is 0.70. At 4 bits qmax is 7, so the scale is 0.70 / 7 = 0.1 (stored as a 16-bit float it is 0.0999756).
2. Divide and round: 3.2 gives 3, −1.0 gives −1, 0.4 gives 0, −7.0 gives −7.
3. Multiply back: 0.30, −0.10, 0 and −0.70. The gaps are 0.02, 0, 0.04 and 0.
4. At 3 bits qmax is 3, so the scale is 0.70 / 3 = 0.233. The codes are 1, 0, 0 and −3, and the weights come back as 0.23, 0, 0 and −0.70. The gaps are 0.09, 0.10, 0.04 and 0.
Each cell below is the code, then the weight it comes back as.
| weight | 4 bits | 3 bits |
|---|---|---|
| 0.32 | 3, 0.30 | 1, 0.23 |
| −0.10 | −1, −0.10 | 0, 0 |
| 0.04 | 0, 0 | 0, 0 |
| −0.70 | −7, −0.70 | −3, −0.70 |
The root-mean-square gap is 0.022 at 4 bits and 0.069 at 3 bits. The weights' own root-mean-square size is 0.389, so the errors are 5.8% and 17.8%. At 3 bits the two small weights vanish: both round to zero.
### The real implementation
`quantize`, `dequantize` and two of the size and speed functions from `code/quantize.py`, with docstrings and input checks left out (`qmax` returns `2 ** (bits - 1) - 1`, and `_blocks` reshapes a tensor into groups):
```python
def quantize(w, bits, group=None):
w = np.asarray(w, dtype=np.float64)
top = qmax(bits)
blocks = _blocks(w, group)
scales = (np.abs(blocks).max(axis=-1, keepdims=True) / top).astype(np.float16)
step = scales.astype(np.float64)
step[step == 0] = 1.0
codes = np.clip(np.rint(blocks / step), -top, top).astype(np.int32)
return codes.reshape(w.shape), scales
def dequantize(codes, scales, group=None):
return (_blocks(codes, group) * scales.astype(np.float64)).reshape(codes.shape)
def weight_bytes(cfg, bits, group=GROUP):
embeddings = embedding_params(cfg)
others = param_count(cfg) - embeddings
total = embeddings * FULL_BITS // 8 + others * bits // 8
if bits != FULL_BITS and group is not None:
total += others // group * SCALE_BITS // 8
return total
def tokens_per_second_bound(bytes_read, bandwidth):
return bandwidth / bytes_read
```
For Llama 3.1 8B, 1,050,673,152 weights are embeddings (the token table and the output matrix) and 6,979,588,096 are not. The model shape comes from a saved copy of its `config.json`, and the weight count is the one from [lesson 18](https://spicyshark.co/learn/model-memory/). The scales add 109,056,064 bytes (109.1 MB): 1.2% of the 8-bit file, 1.9% of the 4-bit file and 2.3% of the 3-bit file.
| bits | per weight | file |
|---|---|---|
| 16 | 16 | 14.96 GiB |
| 8 | 8.125 | 8.56 GiB |
| 4 | 4.125 | 5.31 GiB |
| 3 | 3.125 | 4.50 GiB |
The bits per weight stored count only the quantized weights. Without scales the files would be 8.46, 5.21 and 4.39 GiB.
The speed ceiling divides bandwidth by the bytes read for one token. An untied model such as this one looks up a single row of its token table (4,096 weights, 8,192 bytes) instead of reading all 1,050,673,152 bytes of it, so `bytes_per_token` is the file minus the token table plus one row. That gives 15,009,857,536 bytes at 16 bits, 8,139,325,504 at 8, 4,649,531,456 at 4 and 3,777,082,944 at 3, so the ceilings sit above bandwidth divided by file size: by 7% at 16 bits and 28% at 3 bits. Bandwidths: 3.35 TB/s for the H100 SXM and 4.8 TB/s for the H200 (NVIDIA product pages, which mark the H200 figures as preliminary), and 2,039 GB/s for the A100 80 GB SXM (NVIDIA data sheet), all read 2026-10-19.
| bits | H100 SXM | A100 80 GB SXM |
|---|---|---|
| 16 | 223 | 136 |
| 8 | 412 | 251 |
| 4 | 721 | 439 |
| 3 | 887 | 540 |
On an H200 the ceilings are 320, 590, 1,032 and 1,271 tokens a second. A simpler rule, that every token reads the whole file, gives 209, 365, 588 and 694 on an H100, because it counts the token table in every token. None of these counts the attention cache or anything else the chip reads, so all are ceilings.
### Checked against published files
The sizes are checked against the `.safetensors` files of published copies of Llama 3.1 8B (sizes from the Hugging Face API, read 2026-10-19; both columns are in bytes and the gap is the file minus the formula). The 16-bit file is the base model and the others the Instruct version, which has the same shape. The 4-bit AWQ and GPTQ files appear to hold a 4-bit zero point beside each scale (half a byte a group), so the formula adds it for them.
| copy | file | gap |
|---|---|---|
| 16-bit | 16,060,556,376 | 33,880 |
| 8-bit | 9,084,040,952 | 354,040 |
| AWQ | 5,727,938,576 | 478,144 |
| GPTQ | 5,735,719,888 | 3,278,720 |
For the 8-bit copy the formula uses one 16-bit scale for each output row (1,376,256 scales, 2,752,512 bytes) instead of one for each group, because its card says so. For the AWQ copy it adds those zero points, 27,264,016 bytes. The gap is 0.0083% of the AWQ file and 0.057% of the GPTQ file. For the GPTQ copy the formula also adds a 4-byte column index (called `g_idx`) for each of 1,245,184 columns, 4,980,736 bytes; the remaining 3,278,720 bytes are not accounted for.
### What the accuracy studies measured
The three-model results in Figure 4 come from Table 4 of the AWQ paper (Lin et al., arXiv 2306.00978, version 6), measured as WikiText-2 perplexity; lower is better. Plain rounding in groups of 128 is the paper's "RTN" rows. Figure 4 shows the rise over 16 bits.
| Llama-2 | 16 bits | 4 bits | 3 bits |
|---|---|---|---|
| 7B | 5.47 | 5.73 | 6.66 |
| 13B | 4.88 | 4.98 | 5.52 |
| 70B | 3.32 | 3.46 | 3.98 |
For Llama-2 7B the same table gives GPTQ 5.69 and AWQ 5.60 at 4 bits, and 6.43 and 6.24 at 3 bits, against 5.73 and 6.66 for plain rounding. The paper's plain-rounding baseline may use a different scale rule from this lesson's symmetric code (for example, one that also shifts by a zero point), so treat the match as close, not exact.
Which weights get extra care matters. In the paper's Table 1, for OPT-6.7B at 3 bits in groups of 128, plain rounding scores 23.54 against 10.86 at 16 bits. Keeping 1% of the channels at 16 bits brings it to 11.39 when the channels are chosen by the size of the numbers flowing through them, and only to 22.37 when chosen by the size of their weights and 24.23 when chosen at random. A related finding, in the LLM.int8() paper (Dettmers et al., arXiv 2208.07339), is about activations rather than weights: from about 6.7 billion parameters, models carry about 0.1% of feature dimensions up to 20 times larger than the rest, and setting them to zero raises perplexity by 600% to 1,000%. On Llama-2 7B GSM8K in the AWQ paper's Table 8, 4-bit plain rounding scores 11.07 against 13.87 at 16 bits, GPTQ 12.13 and AWQ 13.57.
For the Llama 3.1 8B Instruct copies, each model card reports its own 16-bit run, so compare only within a card. The average is the Open LLM Leaderboard version 1 average:
| benchmark | 16 bits | 4 bits |
|---|---|---|
| average (v1) | 74.3 | 73.5 |
| MMLU, 5-shot | 68.3 | 66.9 |
| GSM-8K | 82.8 | 82.9 |
| TruthfulQA | 54.5 | 52.8 |
| MMLU-Pro | 30.8 | 28.8 |
| IFEval | 77.9 | 76.3 |
Both copies were made with GPTQ. The 4-bit copy uses groups of 128 and rounds only the linear layers inside the transformer blocks. The 8-bit copy uses one scale for each output row. Its card gives 74.05 and 74.12 for the average, 80.47 and 80.48 for Hellaswag, 78.06 and 77.51 for Winogrande and 54.48 and 54.41 for TruthfulQA, as 16-bit and 8-bit.
### Proof it works
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -B -m pytest -q -p no:cacheprovider code/
```
The tests check these claims. The hand-worked group above comes out as the table says, at 4 and at 3 bits. Ties round to even. Codes stay inside their range and the largest weight survives, at 8, 4 and 3 bits, with one scale and with groups. A group of zeros does not divide by zero, even with floating-point errors set to raise, and a group size that does not divide the row is refused. Error rises as bits fall, and groups beat one scale. One weight of 1.0 pushes the shared-scale error above 90% and moves the per-group error by less than half a point, and the four errors the lesson quotes (19.5%, 11.8%, 99.5% and 11.9%) are pinned to the seed. Stored bits per weight come to 4.125, 8.125 and 3.125. The sizes agree with [lesson 18](https://spicyshark.co/learn/model-memory/)'s weight count, with the AWQ file to within 0.01% and with the 8-bit file between the formulas with no scales and with one scale a group. A token reads one row of the token table, and the speed ceilings come out at 223, 412, 721 and 887.
The widget is a separate copy of the arithmetic in JavaScript, so its tests keep it honest. Its data block must equal what `embed_widget.py` writes from the Python constants; every element its script looks up must exist; it must stay under 40 KiB and make no network call. A parity test runs the widget's number code under Node and compares it with Python for 567 quantizer cases (63 weight vectors, 3 bit widths, 3 group settings), for 16-bit rounding of 406 values, for ties to even, and for the sizes and speeds on each card. A render test runs the page script under Node and reads the three readouts: 5.31 GiB, 721 tokens a second and 4.7% on opening, 19.1% with the big weight, and 34.7% after choosing "One for all", with size and speed unchanged.
What the tests do not show: that a trained matrix behaves like a bell curve, or that a real server reaches the ceiling. To check the second, time one user's decoding and compare it with the ceiling.
### Where it is used
- **Published 4-bit copies.** The AWQ and GPTQ copies of Llama 3.1 8B Instruct from hugging-quants are the files checked above, and the 4-bit copy from Red Hat's model hub uses groups of 128, as its card states.
- **Published 8-bit copies.** The Red Hat 8-bit copy keeps the output matrix at 16 bits (its config ignores only `lm_head`) and uses a scale for each output row.
- **A real example from this site.** The [dynamic GGUF copy of Qwen3.8-27B-OBLITERATED](https://huggingface.co/spicysharkco/Qwen3.8-27B-OBLITERATED-DYN-GGUF) gives each group of matrices its own number of bits, more where measurement shows rounding hurts most. At the same 16.81 GB as the upstream 4-bit Q4_K_M file, its next-word probabilities drift less from the 16-bit model's: mean KL divergence 0.0195 against 0.0348. The [Research page](/research/) has the full table.
- **The papers.** GPTQ (Frantar et al., arXiv 2210.17323) reports that plain rounding collapses at 3 bits where its method stays usable. AWQ (above) picks the channels to protect from the activations. LLM.int8() (above) splits the large dimensions out and keeps them at 16 bits.
## Skills this lesson builds
- Round weights onto a signed grid with a scale for each group, and write the rounding and the dequantizing in numpy.
- Count the stored bits for each weight including scales and predict a quantized file's size, then check the prediction against a published file.
- Measure rounding error on a seeded matrix, show how one large weight changes it, and say why that error is a proxy rather than model accuracy.
- Bound the decoding speed of a model from the bytes read for each token and a card's memory bandwidth.
- Read a published quality claim: which baseline, which benchmark, which rounding method and which weights were rounded.
**Portfolio project:** round a small open model to 8, 4 and 3 bits with plain round-to-nearest in groups, using your own code. Measure the file size, the perplexity on a text file the model has not seen, and the decoding speed on hardware you own. Publish a table that sets each speed against the bandwidth ceiling and each perplexity against the 16-bit original, and say which weights you left at 16 bits.
## Check yourself
1. What does quantization trade away, and what does it buy?
Answer
It gives up exactness: each weight is rounded, so answers can shift a little, and more at 3 bits. It buys a smaller file (Llama 3.1 8B goes from 14.96 GiB to 5.31 GiB at 4 bits) and a higher speed limit, because fewer bytes are read for each token (223 to 721 tokens a second on an H100).
Answer
The scale follows the largest weight in its group. One huge weight stretches a shared scale, so the steps become wider than most weights and they round to zero (in the made-up matrix the error rose from 19.5% to 99.5%). With groups of 128, only the group holding that weight is stretched, and the error stayed at 11.8% to 11.9%.
Answer
The largest size is 0.90 and qmax at 3 bits is 3, so the scale is 0.30. Divide and round: 0.50 / 0.30 = 1.67 gives 2, −0.20 / 0.30 = −0.67 gives −1, 0.07 / 0.30 = 0.23 gives 0, and −0.90 / 0.30 = −3 gives −3. Multiply back: 0.60, −0.30, 0 and −0.90. The gaps are 0.10, 0.10, 0.07 and 0.
Answer
Each weight takes 4 + 16 / 128 = 4.125 bits, so the file is 7,000,000,000 x 4.125 / 8 = 3,609,375,000 bytes, which is 3.61 GB. The ceiling is 2,039 / 3.609 = about 565 tokens a second. It is a ceiling because the chip also reads other things and may not reach its full bandwidth.
Answer
The error measures how far rounded numbers are from the originals in a made-up matrix. It is a proxy, not accuracy. Accuracy depends on which weights matter and has to be measured on the finished model. For real models, the published 4-bit perplexity rose by 2.0% to 4.8% with plain rounding in groups of 128, and the average benchmark score of a 4-bit GPTQ copy of Llama 3.1 8B Instruct fell by 0.8 points.