LoRA Under the Hood โ Why 7 Billion Weights Fit on Your GPU
Lesson 3: LoRA Under the Hood โ Why 7 Billion Weights Fit on Your GPU
You cannot reason about memory budgets, rank choices, or why your 8GB card can fine-tune a 7B model without understanding what LoRA actually does to a weight matrix. Fortunately the maths is one equation.
First, why full fine-tuning is so brutal
Training a model means storing more than the model. For a 7B-parameter model in half precision (bf16, 2 bytes per number), a full fine-tune needs:
gradients (bf16) 7e9 ร 2 bytes = 14 GB
Adam optimizer state 7e9 ร 2 tensors ร 4 bytes = 56 GB
โโโโโโโโโโโโโโโโโโโโโ
static footprint โ 84 GB
plus activations, buffers, fragmentation + 10โ40 GB
Adam keeps two fp32 numbers (a running mean and a running variance) for every single trainable parameter. That line item alone is four times the size of the weights themselves. This is why a 7B full fine-tune is conventionally run across eight 80GB A100s with sharding (ZeRO/FSDP) โ or, on newer hardware, a few H100s. The optimiser state, not the model, is what forces the cluster.
What LoRA actually does
For a targeted weight matrix Wโ of shape d ร k (frozen), LoRA adds a trainable low-rank update:
Wโ : d ร k, frozen โ never updated, stored in bf16 or 4-bit NF4
A : r ร k, trainable โ initialised randomly (Gaussian)
B : d ร r, trainable โ initialised to zero, so ฮW starts at exactly 0
r : the rank โ the inner dimension you choose (8, 16, 32, 64โฆ)
ฮฑ/r : the scaling factor that keeps update magnitude stable as r changes
B initialised to zero matters more than it looks: at step 0 the model's behaviour is exactly the base model's. Training only ever departs from a known-good starting point, which is why LoRA is so much more forgiving than full fine-tuning.
The trainable-parameter maths, concretely
For a d ร k matrix, LoRA trains r ร (d + k) parameters instead of d ร k. Take Llama-class 7B (hidden size 4096, MLP intermediate 11008, 32 layers) targeting all seven linear projections per layer at r = 16:
| Module | Base shape | Base params | LoRA params (r=16) |
|---|---|---|---|
| q_proj / k_proj / v_proj / o_proj | 4096 ร 4096 | 16,777,216 each | 131,072 each |
| gate_proj / up_proj / down_proj | 4096 ร 11008 | 45,088,768 each | 241,664 each |
| Per layer (7 modules) | โ | 202,389,504 | 1,249,280 |
| ร 32 layers | โ | 6,476,464,128 | 39,976,960 โ 40M |
That's ~0.6% of the model trainable. Note how the MLP projections dominate โ which is exactly why attention-only LoRA recipes lose quality: adapting just q_proj and v_proj touches about 8% of the parameters that full-rank adaptation of all linear layers would.
QLoRA: the 4-bit trick that made this mainstream
LoRA still needs the frozen weights resident, and at bf16 a 7B base is 14 GB. QLoRA (Dettmers et al., 2023) adds three things:
| Ingredient | What it does | Effect |
|---|---|---|
| NF4 quantisation | Stores frozen weights in 4-bit NormalFloat โ a data type whose levels are placed at the quantiles of a normal distribution, which fits normally-distributed weights better than plain 4-bit integers. | 14 GB โ ~3.5 GB for a 7B base |
| Double quantisation | Quantises the quantisation constants themselves. | Saves a further ~0.4 GB per billion params |
| Paged optimisers | Pages optimiser state to CPU memory on spikes instead of OOM-ing. | Tolerates longer sequences during a batch |
Because the frozen base is 4-bit and only the adapters are trained in bf16, gradients flow through the dequantised forward pass while the stored weights stay compressed. That is how a 7B model trains inside a single 16 GB free-tier GPU, and how the QLoRA paper demonstrated a 65B model on one 48 GB card.
LoRA (bf16 base) โ best quality, needs ~2ร the base model in VRAM before activations.
QLoRA (4-bit base) โ ~4ร less VRAM, ~1% quality cost, some step-time cost. The default when hardware is the constraint.
Full fine-tune โ only when you have the cluster and need the last few percent.
Target modules, and the honest default
# The classic 2023 recipe โ adapt attention only
target_modules = ["q_proj", "v_proj"]
# The modern default โ adapt every linear layer (attention + MLP)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"]
# In practice you rarely enumerate them: both Unsloth and PEFT accept
# a wildcard and figure it out per architecture.
target_modules = "all-linear""all-linear" is what you should reach for. It roughly triples the trainable parameters versus q,v only while adding a fraction of a percent to the total โ and the quality gap is consistently worth it for behaviour-changing tasks.
๐บ Watch:
- LoRA & QLoRA Fine-tuning Explained In-Depth โ Mark Hennings (196k views). Works through exactly this decomposition with diagrams and code.
- Fine-tune your own LLM in 13 minutes, here's how โ David Ondrej. A real QLoRA run start to finish, useful for seeing how small the rank numbers actually are in a working script.
๐ง Knowledge Check
1. Why does a full fine-tune of a 7B model need ~84 GB of non-activation memory, while QLoRA needs a small fraction of that?
2. In W = Wโ + (ฮฑ/r)ยทBยทA, why is B initialised to zero?
3. You double the LoRA rank from 16 to 32 but keep alpha at 16. What happens to the effective update scale?