Lesson 1: Why Quantization Exists

A local LLM is a pile of numbers — billions of weights (parameters) organized into matrices. Qwen3.8-27B has about 27 billion of them. How much space they take depends on how precisely each number is stored:

Precision	Bytes per weight	Qwen3.8-27B size	Notes
fp32	4	~108 GB	Training precision, never shipped
bf16 / fp16	2	~54.7 GB	Original release quality
8-bit	1	~29 GB (Q8_0)	Nearly lossless
4-bit	0.5	~16.5 GB (Q4_K_M)	The local sweet spot
2-bit	0.25	~6-10 GB (IQ2/UD-Q2)	Cramming mode, real quality loss

The bf16 file for Qwen3.8-27B is 54.7 GB (two ~50 GB + ~4.7 GB shards in the unsloth repo). A typical consumer GPU has 8-24 GB of VRAM. Something has to give — and what gives is quantization: storing each weight with fewer bits, plus a small per-group scale factor to keep the values roughly right.

Key idea: quantized models are the same model, with the numbers rounded to fewer levels. You are not downloading a worse model — you are downloading a compressed copy of the same one. The art is compressing it without breaking it.

How much quality do you lose?

The standard yardstick is perplexity (lower = better prediction). Measured on a 7B-class model against the fp16 baseline:

Quant	Perplexity vs fp16	Verdict
Q8_0	+0.03%	Indistinguishable
Q6_K	+0.13%	Best quality/size ratio
Q5_K_M	+0.39%	Balanced
Q4_K_M	+1.68%	Recommended default
Q3_K_M	+6.07%	Noticeable, small models only
Q2_K	+15.3%	Last resort

These exact numbers shift per model family, but the shape of the curve never does: 8-bit is basically free, 6-bit and 5-bit are nearly free, 4-bit costs a little, and below 3-bit quality falls off a cliff.

The one rule that matters more than any quant choice

Rule of thumb: pick the biggest model that fits your hardware, then the highest quant that still fits. A 27B at 4-bit beats a 7B at 8-bit on almost every real task. Quantization is how you fit the big model — not a way to polish a small one.

Bonus: speed and cost

Fewer bits means less memory bandwidth per token — and memory bandwidth is the bottleneck on local GPUs. A 4-bit model doesn't just fit; it also runs meaningfully faster than the same model at 8-bit, and leaves VRAM free for a longer context (the KV cache). Quantization is the rare win-win: it makes models smaller, faster, and cheaper, at the price of a few percent of quality.

🧠 Knowledge Check

1. Qwen3.8-27B in bf16 is about how large?

2. Which statement is correct?

3. Perplexity increase at Q4_K_M (vs fp16) is roughly:

Lesson 1: Why Quantization Exists