Lesson 2: How Quantization Actually Works

Lesson 2: How Quantization Actually Works

A 4-bit number can only hold 16 distinct values (2⁴). If you rounded every weight to one of 16 values, the model would collapse — weights vary across huge ranges. The trick is grouped quantization with scaling:

  1. Split the weight matrix into blocks (typically 32-128 weights each).
  2. Record one scale per block — a normal float that captures that block's magnitude (min/max or similar).
  3. Store each weight in the block as a small integer relative to the scale (e.g. 0-15 for 4-bit).
  4. At inference, each value is reconstructed as scale × int on the fly.

The scale is the hidden cost: a 4-bit quant is really ~4.85 bits per weight once you count the scales and metadata. That number — total bits divided by parameter count — is bits per weight (bpw), and you'll see it in filenames like 3.69bpw.

One block of 16 weights, 4-bit quantized fp16 weights: -0.7 0.3 1.9 -2.1 0.4 1.1 -1.8 2.3 0.9 -0.2 1.5 -2.6 0.6 0.1 -1.4 2.0 scale = 2.6 (block magnitude) → each weight becomes an int 0..15 4-bit ints: 3 5 12 1 6 9 2 14 8 4 11 0 7 6 3 13 + scale (1 float) Reconstruct: value ≈ scale × (int / 15 - 0.5). Slight rounding error per weight — that error is "quantization error".

Why K-quants say "K" and why they're better

The old Q4_0/Q4_1 scheme used one simple scale per block. K-quants (the K in Q4_K_M) use k-means-style codebooks and mixed precision: attention and output layers get more bits, feed-forward layers get fewer. Since attention weights do the heavy lifting in reasoning, this buys quality at almost no size cost. The second letter is the size tier: S (small — fewer tensors upgraded), M (medium — the default), L/XL (large — more tensors at higher precision).

Weights vs activations: W4A16

Weights are static — you can spend hours calibrating them. Activations (the data flowing through the network at inference time) are dynamic and full of wild outliers, so they're much harder to quantize. Most local formats are W4A16: 4-bit weights, 16-bit activations. When you see W8A8 or W4A8, both weights and activations are quantized — that's what the big serving engines (vLLM, TensorRT) do for throughput.

Not all errors are equal — calibration is the 2024-2026 revolution

Roughly 1% of weights are "outliers" that dominate the model's behavior. Blind rounding destroys them; calibrated quantization runs a sample of text through the model, measures which weights matter, and protects them:

  • imatrix — an importance matrix from calibration data, used when quantizing GGUFs.
  • AWQ — activation-aware: scales the important channels before rounding.
  • GPTQ — error-correction pass that nudges remaining weights to compensate for rounded ones.
  • QAT — the model itself is fine-tuned while simulating quantization, so it learns to be robust to it.
What breaks first at low bits: math, code, exact formatting, instruction-following, and long-chain reasoning — anything precision-sensitive. Casual chat and creative writing survive surprisingly deep quantization. If a Q2 model writes decent prose but fails every arithmetic question, that's normal, not a broken file.

The deep lesson

Quantization error compounds with model size and task difficulty. A well-calibrated 4-bit quant of a 27B model is often smarter than an 8-bit quant of a 7B model — bigger base + decent compression beats smaller base + lossless. That's the entire game of local LLMs.

🧠 Knowledge Check

1. In grouped quantization, what does the scale do?

2. The 'K' in Q4_K_M refers to:

3. W4A16 means:

Further Reading