Hyperparameters, VRAM and Time — The Numbers

Lesson 6: Hyperparameters, VRAM and Time — The Numbers

Three questions decide whether your project is feasible: what do I set?, will it fit?, and how long will it take? This lesson answers all three with real figures — the VRAM table below is Unsloth's own published minimums, and the hyperparameter defaults are the ones their documentation recommends after "hundreds of research papers and experiments."

Does it fit? The VRAM table

Minimum VRAM required, by model size and method. QLoRA is 4-bit, LoRA is 16-bit. These are absolute minimums — real runs often need more, especially at longer context.

Model size	QLoRA (4-bit)	LoRA (16-bit)	What that means in practice
3B	3.5 GB	8 GB	Anything with a GPU. Even a 6 GB laptop card.
7B	5 GB	19 GB	Free Colab T4 comfortably. LoRA needs a 24 GB card.
8B	6 GB	22 GB	The sweet spot. 8B QLoRA is the recommended first project.
9B	6.5 GB	24 GB	Fits a 4090 / L4 / T4 in QLoRA.
11B	7.5 GB	29 GB	—
14B	8.5 GB	33 GB	Still fits a 16 GB T4 in QLoRA, tighter at long context.
27B	22 GB	64 GB	Needs a 24 GB card (3090/4090) or an A6000 for LoRA.
32B	26 GB	76 GB	32 GB card, or A100 80 GB for 16-bit.
40B	30 GB	96 GB	—
70B	41 GB	164 GB	QLoRA on a single A100 80 GB. LoRA needs multi-GPU.
81B	48 GB	192 GB	—
90B	53 GB	212 GB	—
405B	237 GB	950 GB	A cluster. Not a hobby project.

Read the table as a budget, not a promise. The dominant variable it does not show is sequence length. Attention memory grows roughly with the square of context length, so a 27B QLoRA run that fits in 22 GB at 2,048 tokens will not fit at 8,192. If you OOM, the fix order is: reduce batch size to 1, then reduce max_seq_length, then raise gradient_accumulation_steps to keep the effective batch size where you want it. Unsloth's docs call an oversized batch size the most common cause of OOM.

Matching hardware to ambition

Hardware	VRAM	Comfortable workload (QLoRA, r=16, 2k ctx)
Free Google Colab T4	16 GB	8B model, up to ~14B. The default starting point.
Kaggle (T4 ×2 or P100)	16 GB ×2	Same, with more weekly hours than Colab's free tier.
RTX 3090 / 4090	24 GB	Up to 27B QLoRA, or 8B at 16-bit LoRA.
A100 40 GB	40 GB	70B QLoRA at moderate context.
A100 / H100 80 GB	80 GB	70B QLoRA at long context, 32B at 16-bit.
Multi-GPU (2–8×)	160 GB+	Full fine-tuning territory, or 405B QLoRA.

Two Unsloth-specific facts that change these numbers: use_gradient_checkpointing="unsloth" cuts memory a further ~30%, and the framework supports roughly 4× longer context than the standard stack at the same VRAM. So "minimum" here is genuinely a floor, not a comfortable target.

The hyperparameter defaults that actually work

Parameter	Recommended	Why / how it moves quality
`learning_rate`	2e-4	The recommended starting point for LoRA and QLoRA. Full range 2e-4 to 5e-6. For reinforcement learning (DPO, GRPO), use ~5e-6 instead. Full fine-tuning wants lower rates generally.
`num_train_epochs`	1–3	Beyond 3 epochs on instruction data, returns diminish and overfitting rises. Start at 1 and check your eval set before adding more.
`r` (rank)	16	Choose 8, 16, 32, 64 or 128. Small and complex datasets want higher rank; rank usually lands between 4 and 64. Unsloth suggests 8 or 16 for fast runs. Too large overfits.
`lora_alpha`	= r, or 2r	Scaling is `alpha / r`, so keep that ratio at 1 or 2. Alpha = r is the reliable baseline; alpha = 2r learns more aggressively. This is the cheapest knob to adjust after a run that over- or under-cooks.
`lora_dropout`	0	Zero is the optimised path and recent research suggests dropout is an unreliable regulariser on the short runs typical of fine-tuning. Raise to ~0.1 only if you suspect overfitting.
`weight_decay`	0.01–0.1	Raises regularization. 0.01 is the recommended minimum; don't go large.
`warmup_steps`	5–10% of steps	Gradually raises the learning rate at the start so the first update isn't destructive.
`lr_scheduler_type`	linear / cosine	How the rate decays over the run. Either is fine; both are standard.
`per_device_train_batch_size`	2	Primary driver of VRAM. Raise only if you have memory to spare — and note that larger batches can actually be slower due to padding on short examples.
`gradient_accumulation_steps`	4–8	Primary driver of training time. Simulates a larger batch without more memory. Prefer raising this over raising batch size.
effective batch size	16	`batch_size × gradient_accumulation_steps`. A target of 16 is a good default for most fine-tuning (e.g. 2 × 8). This is the number that actually governs stability.
`target_modules`	all 7	`q,k,v,o,gate,up,down`. Targeting all major linear layers is what lets LoRA match full fine-tuning.
`max_seq_length`	2048	2048 is the recommended value for testing. Doubling it roughly squares attention memory.
`random_state` / seed	3407	Reproducibility. Without a fixed seed you cannot tell a real improvement from run-to-run noise.
`bias`	"none"	Leaves bias terms frozen: faster, less memory, negligible quality impact.

🪙 Token angle: two settings dominate your GPU bill, and neither is the one people fiddle with. Effective batch size (raise it via gradient accumulation — free) and epoch count (1–3, and 1 is often right). Going from 1 epoch to 3 epochs triples your training cost for a gain your eval set can usually detect as negative. Train short, evaluate, and only extend the run when the curve says it is still improving.

How long will it take?

Training wall-clock time is throughput arithmetic, and you can compute it before you spend a cent:

total_tokens   = examples × avg_tokens_per_example × epochs
training_time  = total_tokens ÷ throughput_tokens_per_second

# Worked example: 1,000 examples averaging 500 tokens, 1 epoch
total_tokens   = 1000 × 500 × 1 = 500,000 tokens

# The trainer prints iterations/sec and tokens/sec while running.
# Read that number at step 10 and extrapolate — don't guess.

Non-padding throughput varies enormously with model size, method, sequence length and GPU generation, so rather than trust a table, run a 20-step smoke test (max_steps=20) and read the reported speed. That takes about two minutes and gives you an exact answer for your hardware, your model and your sequence length — a number no blog post can give you.

What the speed test will show you qualitatively: QLoRA is slightly slower per step than 16-bit LoRA but uses ~4× less memory, so it is nearly always the right call on consumer hardware. Unsloth's kernel-level optimisations make both substantially faster than the equivalent plain Hugging Face run. And the two settings that move wall-clock the most are gradient_accumulation_steps (more accumulation = proportionally more time per step) and max_seq_length (attention cost grows faster than linearly).

The time nobody budgets for

The honest headline: a first fine-tune is a data project that happens to end in a training run. Budget one to three weeks for a serious first attempt, of which the GPU is busy for a tiny fraction. If you find yourself waiting on training more than working on data, something has gone wrong with your priorities.

Plan for at least three iterations. Run one almost never ships. The realistic loop is: train → evaluate → discover the dataset has 400 near-duplicate rows and a chat-template mismatch → fix → retrain. If your budget assumes one run, it is wrong by a factor of three. The good news: because iteration two and three reuse the same pipeline, they cost a fraction of the first.

📺 Watch:

Fast Fine Tuning with Unsloth — Matt Williams. Demonstrates the speed and memory claims on real hardware.
Fine-tune your own LLM in 13 minutes, here's how — David Ondrej. Useful for calibrating how short an actual training run is.
How to Fine-Tune any AI Model Locally (FULL Tutorial) — Tech With Tim. Includes the OOM-and-fix cycle this lesson warns about.

🧠 Knowledge Check

1. You have a free 16 GB Colab T4 and want to fine-tune an 8B model. Which method and why?

2. You set per_device_train_batch_size=2 and gradient_accumulation_steps=8. What is your effective batch size, and which setting would you raise first to save VRAM?

3. Where does most of the wall-clock time in a realistic first fine-tuning project actually go?